2 spouts one Storm Bolt

Question

I want to create a topology with 2 kafkaSpouts with 2 different topics and merge these 2 spouts into one stream based on sourceComponent in a bolt.

public class Topology {
private static final String topic1 = Real2
private static final String topic2 = Real1


public static void main(String[] args) throws AlreadyAliveException,
        InvalidTopologyException, IOException {

    BasicConfigurator.configure();

    String zookeeper_root = "";
    SpoutConfig kafkaConfig1 = new SpoutConfig(localhost:2181, topic1,
            zookeeper_root, "Real1KafkaSpout");

    SpoutConfig kafkaConfig2 = new SpoutConfig(localhost:2181, topic2,
            zookeeper_root, "Real2KafkaSpout");


    kafkaConfigRealTime.scheme = new SchemeAsMultiScheme(new StringScheme());
    kafkaConfigRealTime.forceFromStart = true;

    kafkaConfigHistorical.scheme = new SchemeAsMultiScheme(
            new StringScheme());
    kafkaConfigHistorical.forceFromStart = true;



    TopologyBuilder builder = new TopologyBuilder();

    builder.setSpout("Real1", new KafkaSpout(
            kafkaConfig1), 2);
    builder.setSpout("Real2", new KafkaSpout(
            kafkaConfig2), 2);

    builder.setBolt("StreamMerging", new StreamMergingBolt(), 2)
            .setNumTasks(2).shuffleGrouping("Real1")
            .shuffleGrouping("Real2");



    Config config = new Config();
    config.put("hdfs.config", yamlConf);

    config.setDebug(false);
    config.setMaxSpoutPending(10000);

    if (args.length == 0) {
        LocalCluster cluster = new LocalCluster();

        cluster.submitTopology("Topology", config,
                builder.createTopology());
        cluster.killTopology("Topology");
        cluster.shutdown();
    } else {

        StormSubmitter.submitTopology(args[0], config,
                builder.createTopology());
    }

    try {
        Thread.sleep(6000);
    } catch (InterruptedException ex) {
        ex.printStackTrace();
    }

}

}

In Bolt Execute Method I am doing

public void execute(Tuple input, BasicOutputCollector collector) {
    String id = input.getSourceComponent();
    System.out.println("Stream Id in StreamMergingBolt is " + "---->" + id);

    }

So I want to store into separate files tuples coming from each stream That is I want to store tuples for Real1KafkaSpout to file1 and Real2KafkaSpout to file2 . How can I do this I was struck at this point

Answer 1

You could do it as follows:

public void execute(Tuple input, BasicOutputCollector collector) {
    String id = input.getSourceComponent();
    if(id.equals("Real1")) {
        // store into file1
    } else {
        assert (id.equals("Real2");
        // store in file2
    }
}

You would open both files in Bolt.open(...) .

However, I am wondering why you want to do this using a single topology. If you only write data from Kafka source one into file one and data from Kafka source two into file two, you could simple create two topologies... (of course, you only program it once, and just configure it differently for both cases).

Answer 2

Wired result when I am doing this by the following code

public void execute(Tuple input, BasicOutputCollector collector) {
String id = input.getSourceComponent();
if(id.equals("Real1")) {
   String json = input.getString(0);
  //writetoHDFS1(json)
   } else {
      assert (id.equals("Real2");
    String json = input.getString(0);
 //writetoHDFS2(json)
  }
 }