Frameworks for creating asynchronous streaming API

Question

The architecture of our application consists of several modules. The modules can run as a single process, or separately on a different server. We are using REST for the interaction between modules when they are on different servers. Now we need to process streaming data between modules. One module sends a request -- another module asynchronously sends back chunks of data (objects). We have tried to use KryoNet and Apache Mina. We have selected the last one and in general everything works. But the solution has several problems, and there is a feeling that we reinvent the wheel.

Maybe there is ready framework for creating asynchronous API to transmit streaming data and that support several transports and built-in serialization:

• local -- when the modules / services interact within a single process
• netty or analog -- when the modules interact with each other on different machines
• REST -- to interact with modules over HTTP

Something like elasticsearch Java API -- all operations can be performed asynchronously, through the network, locally or via REST. Is there a ready-made frameworks for the creation such API?

We are using Scala 2.10 and Java.

Answer 1

How about finagle? https://github.com/twitter/finagle

It doesn't cover all your needs out of the box, but it's a very nice and extensible framework and might provide a good base to build upon.

And you can see an example of doing a streaming server using finagle: https://github.com/twitter/finagle/blob/master/finagle-example/src/main/scala/com/twitter/finagle/example/stream/StreamServer.scala

Answer 2

您当然也应该看看Akka IO： http : //doc.akka.io/docs/akka/snapshot/scala/io.html

Answer 3

I would suggest Scramjet . It will work with your local and analog cases, but it will simply shine when given a task with multiple REST API's in the pipeline - in this case it scales logarithmic while other streaming systems scale exponentially (!). Here's a benchmark scramjet-benchmark on GitHub

It can be programmed with a chain of simple lambdas and is based on node.js with all the plethora of asynchronous modules you'll be able to run all your REST services in a single line (I suggest you tried request-promise). A simple case looks like this:

const scramjet = require("scramjet");
const request = require("request-promise");

process.stdin.pipe(new scramjet.StringStream())
    .split("\r?\n")
    .map((uri) => request.get("https://example.com/" + uri))
    .filter((data) => data.status === 200 && data.articleUrl)
    .map((data) => request.get("http://somesite.org/" + data.articleUrl)
    .accumulate((data) => database.put(data.key, data))
    .then(() => console.log('finished'))

And that's it for a software that calls two API's and writes the results to database.