Working with s3 files into Postgres effciently

Question

I am currently using AWS S3 as a storage for many json files (2 million and counting). I want to put all of these files inside a db, in a Postgres RDS.

I am currently using AWS Lambda which is parsing the files, and it is significantly slower than running it locally. In addition, the work of running the script and installing external modules in Python is pretty terrible with lambda.

Is there a quicker and more efficient way to work with S3 files, parse them and put them in Postgres without the need to download them?

It needs to run on every new file (that's why I chose lambda) and it needs to be divided to couple of tables, so it's not just putting the files as-is (the script already takes the file and parses it to the right tables).

Answer 1

You can use aws glue. But that will cost you for each job run.