From API responses to PostgreSQL tables

Question

Context

I'm searching for an efficient and robust workflow to dump data from API responses to a PostgreSQL database.

Detailed explanation

I have access to a project RESTfulAPI , to which I can send some requests (Fig. 1) and which gives me responses as JSON data.

Eg when using postman for example, I can build a query using all the parameters that are taken into account by the API, and which may for example looks like:

Fig. 1: Request example in cURL .

The corresponding response looks like a bunch of those:

[
    {
        "id": "128",
        "user": "UUID-981729jqdwm91888r",
        "description": {
            "producer": "John Wayne",
            "title": "Alamo",
            "year": 1960
        },
        "preview": {
            "imageURL": "11231068.jpg"
        }
    },
    {
    ...
    }
]

On the other hand, I've a PostgreSQL database (inside a docker container ) which is already structured to admit the response data. For example, all the individual film features from the previous result have to be recorded in a table 'films' (Fig. 2).

Fig. 2: An example of what the resulting PostgreSQL table would look like.

The thing is, I don't know from where to start as I always queried the API through my web browser or Postman until now.

Summarized , I have to dump API responses in a structured set of PostgreSQL tables as follow;
one API route that is requested --> one JSON response (with multiple features) --> one PostgreSQL table (with multiple rows)
one row corresponding to one of the response features).

Question

How could I technically do that; would it be possible to build API requests directly in SQL (I have the feeling it's a super bad idea) or do I have to use an other language or tool (for security reasons or simply for practicality)?

When searching the web for such thing I almost always found answer to "how to make API calls to a Postgres database?", which is obviously not what I want.

Answer 1

Try to make a service which can

• Call your Rest API and get the data
• Structured the data for DB table
• Then put the data in Database

You can set a time interval/ scheduled time for this job done.

You can use golang to build this service or other language you familiar with.

Answer 2

So if I understand you correctly, you want to take data from an web API and dump them into a database of some form, probably for data processing later.

To my surprise, there does exists such a thing in Postgres to do what you want, though you will need to relax the pure SQL requirement. If you are willing to use PL/PGSQL, there exists this plugin that adds a HTTP client for use in Postgres.

For a one off script and you don't mind working with procedural language extensions in Postgres, I imagine this would be fine. However, sanity wise, a Python script would be safer, simpler and overall easier to develop than the PGSQL. Personally, I would go with the Python route but I am not a database administrator which factors into my preference.

In fact, if you don't care for purity of working with a relational database or are not as comfortable, you could use an ORM like SQLAlchemy to simplify this task and later on have the objects be serialized into Python objects with some upfront work. If you dislike the idea of an ORM, you can also use SQLAlchemy to abstract and handle the database connection and tables setup.

Of course, if you're comfortable with SQL, there exists more barebone libraries that setups a database connection to Postgres and includes support for cursors, prepared statements and what not. For Python, there's psycopg2 ; for JavaScript there is node-postgres