I'm trying to use postgres
CopyManager.copyIn()
for batch inserts. My datasource is a c3p0 ComboPooledDataSource
.
The sql statements are batch with:
dataSource.getConnection().getCopyAPI().copyIn(sql, items); //pseudocode
Now to speed up database inserts even more (hundred GB to import after pre-processing), I'm trying to send the copyIn
command in async threads.
But does this make sense if the database is located on a single disk filesystem? Would this gain performance?
And how can I actually verify that the copyIn is using the connection pool in parallel?
I tried VisualVM
MBeans
screen, where i can see a single PooledDataSource
entry. But how can I know that the pool is used and items are send to DB parallel?
But does this make sense if the database is located on a single disk filesystem? Would this gain performance?
If it's spinning rust it might not, and there's certainly little benefit in much concurrency. For SSDs it'll sometimes produce quite a significant improvement. Depends a lot on the drive.
I tried VisualVM MBeans screen, where i can see a single PooledDataSource entry. But how can I know that the pool is used and items are send to DB parallel?
Look at pg_stat_activity
in the database and see if there are multiple concurrent COPY
commands from the app.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.