Which technology can connect with Cassandra as well as NodeJS?

Question

I am using spark streaming for reading TCP server and then inserting the data into Cassandra , which I have to further push to UI, for pushing I decided to go for NodeJS . But I am not getting any technology which can talk with Cassandra as well as NodeJS. Below is my architecture, and I am not able to find out technology which can replace ? I am also open to change Cassandra with MongoDB if it is possible to directly push data from Mongo to NodeJS. But as of now I am using Cassandra as it have native support for Hadoop .

Answer 1

Take a look at the datastax nodejs-cassandra driver: https://github.com/datastax/nodejs-driver project. This has cassandra row streaming and piping functionality that you could use to push cassandra data into node, process, and then export via websockets per your desired architecture.

leave your stream client open - and this would need to run as a persistent node server handling errors - something like this should pick up new cassandra data:

var streamCassandra = function(){
  client.stream('SELECT time, val FROM temperature WHERE station_id=', ['abc'])
  .on('readable', function () {
   //readable is emitted as soon a row is received and parsed
   var row;
   while (row = this.read()) {
    console.log('time %s and value %s', row.time, row.val);
  }})
 .on('error', function (err) {
  //handle err
  })
 .on('end', streamCassandra);
};

wrap your stream client into a recursive function that calls itself again on('end', streamCassandra). you could also poll the function every x seconds with a setInterval if you dont need that kind of concurrency. One of those approaches should work

Answer 2

Have you checked NiFi?

https://nifi.apache.org/

In your case, you could write your Spark Streaming results to Kafka, HDFS, or even directly to NiFi, but I personally prefer to write to Kafka or some other message queue. From NiFi, you can write to Kafka, and also send requests to your Node JS app if that's what you need. In my case, I'm using Meteor, so just pushing from Kafka to MongoDB automatically refreshes the UI.

I hope it helps.