Dockerized Flask App - Rest API with continually running scripts

Question

I have a dockerized Flask application running locally that consists of several REST API endpoints. It's all working as expected so that when a GET request is performed on one of the endpoints, data is retrieved from the Postgres database and then displayed to the browser as json. Great. The database so far is just test data and now I need to continually update the database with real data.

I have the script that pulls data from the web, and I understand how to add it to the database with post and put requests, but what I don't understand, is how and where to have this script continually running, to where it doesn't interfere with the REST API portion of my server and vice versa, almost as though it's a completely separate entity within the backend.

To do this, would I create an entirely new flask app that runs on it's own server and is continually running the script and adding the scraped data to the database so that the other flask app which contains the API endpoints can access it when needed? I feel as though I am way off here, and any input on the best way to move forward is extremely appreciated. Thank you!

Answer 1

You are not far off at all in my oppinion.

I would say, let your API stand on it's own - as the gateway to your database. In it's own container.

The scraping that you want to do is another process - and you should not mix it into the flask API application. Instead, since you are already in the docker realm here - consider creating another image that will do the scraping for you. This can be a bash script, a python app - it's not important. Just as long as that you can keep it as simple as possible.

You could even consider if you can make that application/script image in such a way, that you could run multiple of them in parallel.

Yes, you will have two images to maintain. But they will each be smaller on their own, and less complex. And, if done right - you can scale the activity up if needed.

Consider the first two statements of the UNIX philosophy :

1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".
2. Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.

Maintainability is king in the game of software development. Big cluttered projects have a hard time surviving in the long run.

After thought: If your project is experimental and you just want to prove some concept - then do that. And don't overthink the design. Too many projects die from that too!

Those are my thought at least.