Dse\Exception\RuntimeException: All connections on all I/O threads are busy

Question

We have a facility in our web app to delete large quantities of data. We do this by paginating through all records found against u_id .

The keys we have are designed for other queries we have in the application - ideally, it would be great to have a primary key for u_id but this would break all our other queries.

The below method works well most of the time, however, after deleting approximately 6-8 million of records, we get:

Dse\\Exception\\RuntimeException: All connections on all I/O threads are busy

We also sometimes get a slightly different error message:

Dse\\Exception\\ReadTimeoutException: Operation timed out - received only 0 responses

You'll notice in the below code usleep(2500000) which pauses the script. This has been our workaround but would be good to get this resolved as Cassandra should be able to handle this number of deletes.

$cluster        = \Dse::cluster()
                    ->withDefaultTimeout(3600)
                      ->withContactPoints(env('CA_HOST'))
                        ->build();

$session        = $cluster->connect(env('CONNECT'));
$options        = array('page_size' => 50);
$results        = $session->execute("SELECT * FROM datastore WHERE u_id = $u_id;", $options);
$future_deletes = array();

while (true) {

    foreach ($results as $result) {

      $future_deletes[] = $session->executeAsync("DELETE FROM datastore WHERE record_id = '" . $result['record_id'] . "' AND record_version = " . $result['record_version'] . " AND user_id = " . $result['user_id']);
      $future_deletes[] = $session->executeAsync("UPDATE data_count set u_count = u_count - 1 WHERE u_id = " . $u_id);

    }

    if( !empty($future_deletes) ){
      foreach ($future_deletes as $future_delete) {
          // we will not wait for each result for more than 5 seconds
          $future_delete->get(5);
      }
      //usleep(2500000); //2.5 seconds
    }

    $future_deletes = array();

    if ($results->isLastPage()) {
        break;
    }

    $results = $results->nextPage();

}

//Disconnect
$session = NULL;

For your reference, here are our tables:

CREATE TABLE datastore (id uuid,
    record_id varchar,
    record_version int,
    user_id int,
    u_id int,
    column_1 varchar,
    column_2 varchar,
    column_3 varchar,
    column_4 varchar,
    column_5 varchar,
PRIMARY KEY((record_id), record_version, user_id)
);
CREATE INDEX u_id ON datastore (u_id);

CREATE TABLE data_count (u_id int PRIMARY KEY, u_count counter);

We are running a server with 8GB RAM.

The version of the DSE driver is 6.0.1.

Thank you in advance!

Answer 1

You need to control, how many "in-flight" requests do you have a the same point of time. There is a limit on number of queries per connection, and number of connections. They are controlled by corresponding functions of the Cluster class (can't find fast enough in PHP docs, but it should be similar to Cluster functions in the C++ driver , because PHP is built on top of C++ driver).