How to do a massive importation of backup files to a Postgresql database

Question

I have a folder with many *.backup files to be imported to a Postgresql database. I've created a bash script to do it:

#!/usr/bin/env sh

echo "Importing data..."

psql -h $DB_HOST -p $DB_PORT -U $DB_USER -c "DROP DATABASE IF EXISTS $DB_DATABASE;"
psql -h $DB_HOST -p $DB_PORT -U $DB_USER -c "CREATE DATABASE $DB_DATABASE;"

cd /App/data/backup

for f in *.backup; do
    echo "- $f"
    pg_restore -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_DATABASE -v "$f"
done

cd -
exit 0

Is there an easier/faster way to do this? Is it possible to import files massively?

Note: this script runs in a docker. Don't worry about the "drop database" statement :)

Answer 1

Assuming that you system has multiple CPUs, you can get some speedup by running the import in parallel. Since pg_restore is usually disk-bound process - the gains will be limited. Network performance will also has big impact.

Two options: using pg_restore -j, and processing multiple files in parallel:

Multiple jobs for pg_restore

for f in *.backup; do
    echo "- $f"
    pg_restore -j4 -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_DATABASE -v "$f"
done

Restore multiple backup files in parallel:

ls *.backup | xargs -P4 -I{} pg_restore -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_DATABASE -v "{}"

You want to experiments with various levels of -P, as it is hard to predict the optimal value.