ArangoDB batch mode doesn't work with Java Driver

Question

I am evaluating ArangoDB using spring batch.

I tried to insert some data and, without batch mode, it works as expected.

However, if batch mode is on, the execution of the program hangs.

I am using arango 2.3.3 and com.arangodb:arangodb-java-driver:[2.2-SNAPSHOT,2.2]

    arangoDriver.startBatchMode();

    for(Account acc : items){
        acc.getRecordHash();
        acc.getIdHash();
        arangoDriver.createDocument("AccountCollection", acc);
    }

    arangoDriver.executeBatch();

Any ideas what I am doing wrong?

Answer 1

I tried to reproduce what you are trying, first of all does the collection "AccountCollection" exist ? If not you would get an error in the batch result but still the program should not hang. i created a unittest :

@Test
  public void test_StartCancelExecuteBatchMode() throws ArangoException {

    driver.startBatchMode();

    ArrayList<Account> items = new ArrayList<Account>();
    items.add(new Account());
    items.add(new Account());
    items.add(new Account());
    items.add(new Account());

    for(Account acc : items){
      acc.getRecordHash();
      acc.getIdHash();
      driver.createDocument("AccountCollection", acc, true, false);
    }

    driver.executeBatch();

  }

This works perfectly and returns:

EOB 16:47:01.862 [main] DEBUG com.arangodb.http.HttpManager - [RES]http-POST: statusCode=200 16:47:01.862 [main] DEBUG com.arangodb.http.HttpManager - [RES]http-POST: text=--dlmtrMLTPRT Content-Type: application/x-arango-batchpart Content-Id: request1

HTTP/1.1 202 Accepted Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033214501 Content-Type: application/json; charset=utf-8 Etag: "48033214501" Content-Length: 95

{"error":false,"_id":"AccountCollection/48033214501","_rev":"48033214501","_key":"48033214501"} --dlmtrMLTPRT Content-Type: application/x-arango-batchpart Content-Id: request2

HTTP/1.1 202 Accepted Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033411109 Content-Type: application/json; charset=utf-8 Etag: "48033411109" Content-Length: 95

{"error":false,"_id":"AccountCollection/48033411109","_rev":"48033411109","_key":"48033411109"} --dlmtrMLTPRT Content-Type: application/x-arango-batchpart Content-Id: request3

HTTP/1.1 202 Accepted Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033607717 Content-Type: application/json; charset=utf-8 Etag: "48033607717" Content-Length: 95

{"error":false,"_id":"AccountCollection/48033607717","_rev":"48033607717","_key":"48033607717"} --dlmtrMLTPRT Content-Type: application/x-arango-batchpart Content-Id: request4

HTTP/1.1 202 Accepted Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033804325 Content-Type: application/json; charset=utf-8 Etag: "48033804325" Content-Length: 95

{"error":false,"_id":"AccountCollection/48033804325","_rev":"48033804325","_key":"48033804325"} --dlmtrMLTPRT--

But even when i create intentional errors the application never "hangs". Frank just sent me your source code, i take a look into it. Can you try to find out where the programm is hanging ? is "executeBatch" reached at all ?

Answer 2

I already imported 1.6 Mio documents with your code and still everything works. i guess it might be necessary to monitor your system resources during the import, if anything unusual occurs let us now. Generally speaking it does not seem to be the best practice to perform a one-time bulk import like this using the java api. i would recommend to use arangoimp to import the data directly into the database, this will be much faster. it is documented here

Answer 3

You need to increase the number of open file descriptors. The Mac has a very low limit (256). ArangoDB stores the data in datafiles of a certain chunk size. With large datasets more files are need (and some fd are already used for communication and other stuff).

When ArangoDB runs out of file descriptors, it can neither extend the dataset nor answer new questions. Therefore the import process will hang.