简体   繁体   中英

copy multiple csv files to postgresql tables in one psql copy call

loading a csv file to my postgresql db table using psql utility copy as below;

psql -h $host -p $port -d $db -U $username -c "\copy \"$schema\".\"$table\" from '$datafile' with delimiter as ',' csv"

this works great. However, I need to load many files at the same time. I can run above psql copy command in parallel fashion. Since each call starts a new process (my app is do.net core 2.1 and is running on openshift), increasing the number of file also increases the memory and CPU. Process time also increases. What I actually need is to call psql copy with multiple csv file. I need to create the parallelization on the db, not on the openshift pod.
How can I do this? If this is not possible, what is the most optimum way to load multiple files at a time?

Thankx

cat *.csv | psql -h $host -p $port -d $db -U $username -c "\copy \"$schema\".\"$table\" from STDIN with delimiter as ',' csv"

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM