Heroku/Dash app Python, reading file on Google Cloud Storage

Question

I have a dash web app deployed on Heroku, and it needs to read a.csv file located on Google Cloud Storage. To do this, I give to the app the credentials to access to my Google Cloud account, and then, I can load the file with pandas:

import pandas as pd 
df = pd.read_csv("gs://bucket_name/file_name.csv")

This.csv file is updated regularly but the updates are not taken into account by the application.

The application load the file when deployed but then, it never re-load the file and so, it never takes updates into account until I deploy it again.

Is there a way to force the app read the file each time I refresh the web browser so that each update is taken into account?

Thank you in advance Best regards

Answer 1

I think a decorator would be usefull here. Please do take into account that depending on the size of the file you might get some extra latency as it needs to reload the df each time.
You'll need to decorate each view that needs the df to be reloaded.

Another one would be to set up a specific endpoint that forces a reload of the df and use Heroku's scheduler to call that endpoint. This will remove the extra latency on the other requests, but it will make it show stale data sometimes.

See this short example below...

import Flask
from functools import wraps
import pandas as pd 


app = Flask(__name__)
df = pd.read_csv("gs://bucket_name/file_name.csv")


def reload_df(f):
    @wraps(f)
    def decorated_function(*args, **kwargs):
        global df
        df = pd.read_csv("gs://bucket_name/file_name.csv")
        return f(*args, **kwargs)
    return decorated_function


@app.route("/")
@reload_df
def index():
    return "hello world"


@app.route("/not_reloading_df")
def index():
    return "still using previous DF"


@app.route("/forcereload")
@reload_df
def force_reload():
    return "Reloaded DF"


if __name__ == "__main__":
    app.run(debug=True)