Exporting Cassandra table with DataStax Bulk Loader v1.8 complains about connection pool exhaustion

Question

I run it with these settings:

dsbulk unload -k keyspace -t table
  --connector.csv.delimiter "^"
  --engine.maxConcurrentQueries=4
  --connector.csv.url
  ...

application complains about connection pool exhaustion --> application gets timeouts on connections to cassandra.

• cassandra version 2.13
• cassandra features: 3 nodes - 64 cpu/124Gb ram on each node.

explain on settings dsbulk?

Answer 1

It sounds like your cluster is getting overloaded and cannot handle the unload operation.

You will need to throttle DSBulk to lower the amount of requests. Here are some options you can use as starting points to limit the load on your cluster:

    --driver.advanced.continuous-paging.page-size 1000
    --driver.advanced.continuous-paging.max-pages 10
    --engine.maxConcurrentQueries 5
    --executor.maxPerSecond 5
    --executor.maxInFlight 5

For details on these options, see:

• DSBulk Driver options
• DSBulk Engine options
• DSBulk Executor options

With these settings, it will take a little longer for the unload operation to complete but it will at least minimise the risk of taking down your cluster. Cheers!