从-cluster选项开始时Vert.x性能下降

Question

I'm wondering if any one experienced the same problem.

We have a Vert.x application and in the end it's purpose is to insert 600 million rows into a Cassandra cluster. We are testing the speed of Vert.x in combination with Cassandra by doing tests in smaller amounts.

If we run the fat jar (build with Shade plugin) without the -cluster option, we are able to insert 10 million records in about a minute. When we add the -cluster option (eventually we will run the Vert.x application in cluster) it takes about 5 minutes for 10 million records to insert.

Does anyone know why?

We know that the Hazelcast config will create some overhead, but never thought it would be 5 times slower. This implies we will need 5 EC2 instances in cluster to get the same result when using 1 EC2 without the cluster option.

As mentioned, everything runs on EC2 instances:

• 2 Cassandra servers on t2.small
• 1 Vert.x server on t2.2xlarge

Answer 1

You are actually running into corner cases of the Vert.x Hazelcast Cluster manager.

First of all you are using a worker Verticle to send your messages (30000001). Under the hood Hazelcast is blocking and thus when you send a message from a worker the version 3.3.3 does not take that in account. Recently we added this fix https://github.com/vert-x3/issues/issues/75 (not present in 3.4.0.Beta1 but present in 3.4.0-SNAPSHOTS) that will improve this case.

Second when you send all your messages at the same time, it runs into another corner case that prevents the Hazelcast cluster manager to use a cache of the cluster topology. This topology cache is usually updated after the first message has been sent and sending all the messages in one shot prevents the usage of the ache (short explanation HazelcastAsyncMultiMap#getInProgressCount will be > 0 and prevents the cache to be used), hence paying the penalty of an expensive lookup (hence the cache).

If I use Bertjan's reproducer with 3.4.0-SNAPSHOT + Hazelcast and the following change: send message to destination, wait for reply. Upon reply send all messages then I get a lot of improvements.

Without clustering : 5852 ms With clustering with HZ 3.3.3 :16745 ms With clustering with HZ 3.4.0-SNAPSHOT + initial message : 8609 ms

I believe also you should not use a worker verticle to send that many messages and instead send them using an event loop verticle via batches. Perhaps you should explain your use case and we can think about the best way to solve it.

Answer 2

When you're you enable clustering (of any kind) to an application you are making your application more resilient to failures but you're also adding a performance penalty.

For example your current flow (without clustering) is something like:

client -> 
  vert.x app -> 
    in memory same process eventbus (negletible) ->
    handler -> cassandra
  <- vert.x app
<- client

Once you enable clustering:

client ->
  vert.x app ->
    serialize request ->
      network request cluster member ->
        deserialize request ->
          handler -> cassandra
        <- serialize response
      <- network reply
    <- deserialize response
  <- vert.x app
<- client

As you can see there are many encode decode operations required plus several network calls and this all gets added to your total request time.

In order to achive best performance you need to take advantage of locality the closer you are of your data store usually the fastest.

Answer 3

Just to add the code of the project. I guess that would help.

Sender verticle:

public class ProviderVerticle extends AbstractVerticle {

    @Override
    public void start() throws Exception {
        IntStream.range(1, 30000001).parallel().forEach(i -> {
        vertx.eventBus().send("clustertest1", Json.encode(new TestCluster1(i, "abc", LocalDateTime.now())));
        });
    }

    @Override
    public void stop() throws Exception {
        super.stop();
   }
}

And the inserter verticle

public class ReceiverVerticle extends AbstractVerticle {

    private int messagesReceived = 1;

    private Session cassandraSession;

    @Override
    public void start() throws Exception {

        PoolingOptions poolingOptions = new PoolingOptions()
                .setCoreConnectionsPerHost(HostDistance.LOCAL, 2)
                .setMaxConnectionsPerHost(HostDistance.LOCAL, 3)
                .setCoreConnectionsPerHost(HostDistance.REMOTE, 1)
                .setMaxConnectionsPerHost(HostDistance.REMOTE, 3)
                .setMaxRequestsPerConnection(HostDistance.LOCAL, 20)
                .setMaxQueueSize(32768)
                .setMaxRequestsPerConnection(HostDistance.REMOTE, 20);

        Cluster cluster = Cluster.builder()
                .withPoolingOptions(poolingOptions)
                .addContactPoints(ClusterSetup.SEEDS)
                .build();

        System.out.println("Connecting session");
        cassandraSession = cluster.connect("kiespees");
        System.out.println("Session connected:\n\tcluster [" + cassandraSession.getCluster().getClusterName() + "]");
        System.out.println("Connected hosts: ");

        cassandraSession.getState().getConnectedHosts().forEach(host -> System.out.println(host.getAddress()));

        PreparedStatement prepared = cassandraSession.prepare(
                "insert into clustertest1 (id, value, created) " +
                        "values (:id, :value, :created)");

        PreparedStatement preparedTimer = cassandraSession.prepare(
                "insert into timer (name, created_on, amount) " +
                        "values (:name, :createdOn, :amount)");

        BoundStatement timerStart = preparedTimer.bind()
                .setString("name", "clusterteststart")
                .setInt("amount", 0)
                .setTimestamp("createdOn", new Timestamp(new Date().getTime()));
        cassandraSession.executeAsync(timerStart);

        EventBus bus = vertx.eventBus();

        System.out.println("Bus info: " + bus.toString());
        MessageConsumer<String> cons = bus.consumer("clustertest1");
        System.out.println("Consumer info: " + cons.address());

        System.out.println("Waiting for messages");

        cons.handler(message -> {
            TestCluster1 tc = Json.decodeValue(message.body(), TestCluster1.class);

            if (messagesReceived % 100000 == 0)
                System.out.println("Message received: " + messagesReceived);

            BoundStatement boundRecord = prepared.bind()
                    .setInt("id", tc.getId())
                    .setString("value", tc.getValue())
                    .setTimestamp("created", new Timestamp(new Date().getTime()));
            cassandraSession.executeAsync(boundRecord);

            if (messagesReceived % 100000 == 0) {
                BoundStatement timerStop = preparedTimer.bind()
                        .setString("name", "clusterteststop")
                        .setInt("amount", messagesReceived)
                        .setTimestamp("createdOn", new Timestamp(new Date().getTime()));
                cassandraSession.executeAsync(timerStop);
            }

            messagesReceived++;
            //message.reply("OK");
        });
    }

    @Override
    public void stop() throws Exception {
        super.stop();
        cassandraSession.close();
    }
}