Performance issues : Kafka + Storm + Trident + OpaqueTridentKafkaSpout

Question

We are seeing some performance issues with Kafka + Storm + Trident + OpaqueTridentKafkaSpout

Mentioned below are our setup details :

Storm Topology :

Broker broker = Broker.fromString("localhost:9092")
    GlobalPartitionInformation info = new GlobalPartitionInformation()
    if(args[4]){
        int partitionCount = args[4].toInteger()
        for(int i =0;i<partitionCount;i++){
            info.addPartition(i, broker)
        }
    }
    StaticHosts hosts = new StaticHosts(info)
    TridentKafkaConfig tridentKafkaConfig = new TridentKafkaConfig(hosts,"test")
    tridentKafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme())


    OpaqueTridentKafkaSpout kafkaSpout = new OpaqueTridentKafkaSpout(tridentKafkaConfig)
    TridentTopology topology = new TridentTopology()
    Stream st  = topology.newStream("spout1", kafkaSpout).parallelismHint(args[2].toInteger())
            .each(kafkaSpout.getOutputFields(), new NEO4JTridentFunction(), new Fields("status"))
            .parallelismHint(args[1].toInteger())
    Map conf = new HashMap()
    conf.put(Config.TOPOLOGY_WORKERS, args[3].toInteger())
    conf.put(Config.TOPOLOGY_DEBUG, false)

    if (args[0] == "local") {
        LocalCluster cluster = new LocalCluster()
        cluster.submitTopology("mytopology", conf, topology.build())
    } else {
        StormSubmitter.submitTopology("mytopology", conf, topology.build())
        NEO4JTridentFunction.getGraphDatabaseService().shutdown()
    }

Storm.yaml we are using for Storm is as below :

########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
     - "localhost"
#     - "server2"
# 
storm.zookeeper.port : 2999


storm.local.dir: "/opt/mphrx/neo4j/stormdatadir"

nimbus.childopts: "-Xms2048m"
ui.childopts: "-Xms1024m"
logviewer.childopts: "-Xmx512m"
supervisor.childopts: "-Xms1024m"
worker.childopts: "-Xms2600m -Xss256k -XX:MaxPermSize=128m -XX:PermSize=96m
    -XX:NewSize=1000m -XX:MaxNewSize=1000m -XX:MaxTenuringThreshold=1 -XX:SurvivorRatio=6
    -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
    -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
    -server -XX:+AggressiveOpts -XX:+UseCompressedOops -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true
    -Xloggc:logs/gc-worker-%ID%.log -verbose:gc
    -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m
    -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram
    -XX:+PrintTenuringDistribution -XX:-PrintGCApplicationStoppedTime -XX:-PrintGCApplicationConcurrentTime
    -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal"

java.library.path: "/usr/lib/jvm/jdk1.7.0_25"

supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

topology.trident.batch.emit.interval.millis: 100
topology.message.timeout.secs: 300
#topology.max.spout.pending: 10000

• Size of each message produced in Kafka : 11 KB
• Execution time of each bolt(NEO4JTridentFunction) to process the data : 500ms
• No. of Storm Workers : 1
• Parallelism hint for Spout(OpaqueTridentKafkaSpout): 1

• Parallelism hint for Bolt/Function(NEO4JTridentFunction) : 50

• We are seeing throughput of around 12msgs/sec from Spout.

• Rate of messages produced in Kafka : 150msgs/sec

Both Storm and Kafka are a single node deployment. We have read about much higher throughput from Storm but are unable to produce the same. Please suggest how to tune the Storm+ Kafka + OpaqueTridentKafkaSpout configuration to achieve higher throughput. Any help in this regard would help us immensely.

Thanks,

Answer 1

You should set spout parallelism same as partition count for mentioned topics. By default, trident accept one batch for each execution, you should increase this count by changing topology.max.spout.pending property. Since Trident forces ordered transaction management, your execution method (NEO4JTridentFunction)must be fast to reach desired solution.

In addition,you can play with "tridentConfig.fetchSizeBytes" , by changing it, you can ingest more data for each new emit call in your spout.

Also you must check your garbage collection log, it will give you clue about real point.

You can enable garbage collection log by adding "-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -Xloggc:{path}/gc-storm-worker-%ID%.log" , in worker.childopts settings in your worker config.

Last but not least, you can use G1GC, if your young generation ratio is higher than normal case.

Answer 2

Please set your worker.childopts based on your system configuration. Use SpoutConfig.fetchSizeBytes to increase the number of bytes being pulled into the topology. Increase your Parallelism hint.

Answer 3

my calculations: if 8 Cores and 500MS per bolt -> ~16 Messages/sec. if you optimize the bolt, then you will see improvements.

also, for CPU bound bolts, try Parallelism hint = 'amount of total cores' and increase topology.trident.batch.emit.interval.millis to the amount of time it takes to process entire batch divided by 2. set topology.max.spout.pending to 1.