如何提高ArangoDB中的插入性能

Question

My environment is local machine: ubuntu 12.04 ArangoDB 2.2.4 or 2.2.3 perl driver(ArangoDB) CPU: 3 core 6 threads mem: 3GB

I used save method. Save method is equal to HTTP_GET and HTTP_POST. Execute result is following:

1. one perl process, insert 30000 documents. Avg 700 requests/s. 350 HTTP_GET and 350 HTTP_POST.
2. 10 perl process, insert 30000 documents. Avg 1000 requests/s. 500 HTTP_GET and 500 HTTP_POST.

After running 30 seconds, It'll report HTTP 500 error. I modified perl driver(ArangeDB) code for retrying it. So I can finish this test.

arangodb's log is following when it reported HTTP 500 error.

2014-10-04T14:46:47Z [26642] DEBUG [./lib/GeneralServer/GeneralServerDispatcher.h:403]   shutdownHandler called, but no handler is known for task
2014-10-04T14:46:47Z [26642] DEBUG [./lib/GeneralServer/GeneralServerDispatcher.h:403] shutdownHandler called, but no handler is known for task

I hoped my program can execute avg 3000-5000 requests/s and reduce HTTP 500 error. What's improvements I can use. Thanks!

UPDATE BY 7/10/2014, My insert sample script is following. And I replaced save method by AQL. One perl process, insert 10000 documents, Avg 900 requests/s, 1000 HTTP_POST/s. (no HTTP 500) One perl process, insert 30000 documents, Avg 700 requests/s, 700 HTTP_POST/s. (There will issued HTTP 500, need retry it)

#!/usr/bin/perl

use warnings;
use strict;

use ArangoDB;

my $itdb = ArangoDB->new(
{
    host       => '10.211.55.2',
    port       => 8529,
    keep_alive => 1,
}
);

# Find or create collection
$itdb->create('Node_temp',{isVolatile => JSON::true});
ImpNodes();

sub ImpNodes{

    for(1..30000){
        my $sth = $itdb->query('INSERT {
            "id": "Jony",
            "value": "File",
            "popup": "public",
            "version": "101",
            "machine": "10.20.18.193",
            "text": {
               "Address": ["center","bold","250","100"]
            },
            "menuitem":[
            {
                "value": "New",
                "onclick": "CreateNewDoc",
                "action": "CreateNewDoc"
            }
            ,
            {
                "value": "Open",
                "onclick": "OpenNewDoc",
                "action": "OpenNewDoc"
            },
            {
                "value": "Close",
                "onclick": "CloseDoc",
                "action": "CloseDoc"
            },
            {
                "value": "Save",
                "onclick": "SaveDoc",
                "action": "SaveDoc"
            }]
        } in Node_temp');

        my $cursor = $sth->execute({
            do_count => 1,
            batch_size => 10,
        });
    }
}

And I have modified Arangodb-0.08 for inserting smoothly in Connection.pm. http_post method:

$retries = 100 #for testing
for(1..$retries){
    ( undef, $code, $msg, undef, $body ) = $self->{_http_agent}->request(
        %{ $self->{_req_args} },
        method     => 'POST',
        path_query => $path,
        headers    => $headers,
        content    => $data,
    );
    last if ( $code < 500 || $code >= 600 );
    print "The return code is 5xx,retry http_post!\n";
    print $code, " : " , $msg , " : " , $body;
    select(undef, undef, undef, 3);
}

Answer 1

I strace'd the client program and could verify that a new connection is opened for each request. This causes many system calls to be issued. strace looks like this for each request:

17300 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
17300 ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fffaee760c0) = -1 ENOTTY (Inappropriate ioctl for device)
17300 lseek(3, 0, SEEK_CUR)             = -1 ESPIPE (Illegal seek)
17300 ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fffaee760c0) = -1 ENOTTY (Inappropriate ioctl for device)
17300 lseek(3, 0, SEEK_CUR)             = -1 ESPIPE (Illegal seek)
17300 fcntl(3, F_SETFD, FD_CLOEXEC)     = 0
17300 setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
17300 fcntl(3, F_GETFL)                 = 0x2 (flags O_RDWR)
17300 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
17300 connect(3, {sa_family=AF_INET, sin_port=htons(8529), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
17300 select(8, NULL, [3], [3], {299, 999526}) = 1 (out [3], left {299, 999524})
17300 write(3, "POST /_api/cursor HTTP/1.1\r\nConnection: Keep-Alive\r\nUser-Agent: Furl::HTTP/3.05\r\nHost: 127.0.0.1\r\nContent-Type: application/json\r\nContent-Length: 1032\r\nHost: 127.0.0.1:8529\r\n\r\n", 176) = 176
17300 write(3, "{\"count\":true,\"query\":\"INSERT {\\n            \\\"id\\\": \\\"Jony\\\",\\n            \\\"value\\\": \\\"File\\\",\\n            \\\"popup\\\": \\\"public\\\",\\n            \\\"version\\\": \\\"101\\\",\\n            \\\"machine\\\": \\\"10.20.18.193\\\",\\n            \\\"text\\\": {\\n               \\"..., 1032) = 1032
17300 read(3, 0x15f0af0, 10240)         = -1 EAGAIN (Resource temporarily unavailable)
--
17300 close(3)                          = 0
17300 rt_sigprocmask(SIG_BLOCK, [PIPE], [], 8) = 0
17300 rt_sigaction(SIGPIPE, {SIG_DFL, [], SA_RESTORER, 0x7faa49b221f0}, {SIG_IGN, [], SA_RESTORER, 0x7faa49b221f0}, 8) = 0
17300 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
17300 rt_sigprocmask(SIG_BLOCK, [PIPE], [], 8) = 0
17300 rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER, 0x7faa49b221f0}, {SIG_DFL, [], SA_RESTORER, 0x7faa49b221f0}, 8) = 0
17300 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

I think you will want to avoid establishing and closing connections on each request. This also solves the problem with the OS running out of ports.

To keep the driver from re-opening connections all the time I had to modify FURL as follows:

In line 526 of Furl/HTTP.pm, FURL checks the HTTP response headers it gets from the server. It will read the Connection from the response header there, and compare the header value with the string keep-alive . The problem is that this does not take into account a different case of the response header. ArangoDB returns a header value of Keep-Alive (mind the caps), so FURL does not recognize it properly.

The following change to Furl/HTTP.pm fixes that:

-    if ($connection_header eq 'keep-alive') {
+    if (lc($connection_header) eq 'keep-alive') {

This makes the clients not close the connection after each request and not run out of ports.