具有持久性的HTTP POST请求的非阻塞队列

Question

Before we develop our custom solution, I'm looking for some kind of library, which provides:

Non-blocking queue of HTTP requests

with these attributes:

• Persisting requests to avoid it's loss in case of:
  • network connectivity interruption
  • application quit, forced GC on background app
  • etc..
• Possibility of putting out all these fields:
  • Address
  • Headers
  • POST data

So please, is there anything usable right know, what could save us whole day on developing this?

Right now we don't need any callbacks on completed request and neither saving result data, as there won't be such.

Answer 1

In my humble opinion, a good and straightforward solution would be to develop your own layer (which shouldn't be so complicated) using a sophisticated framework for connection handling, such as Netty https://netty.io/ , together with a sophisticated framework for asynchronous processing, such as Akka http://akka.io/

Let's first look inside Netty support for http at http://static.netty.io/3.5/guide/#architecture.8 :

4.3. HTTP Implementation

HTTP is definitely the most popular protocol in the Internet. There are already a number of HTTP implementations such as a Servlet container. Then why does Netty have HTTP on top of its core?

Netty's HTTP support is very different from the existing HTTP libraries. It gives you complete control over how HTTP messages are exchanged at a low level. Because it is basically the combination of an HTTP codec and HTTP message classes, there is no restriction such as an enforced thread model. That is, you can write your own HTTP client or server that works exactly the way you want. You have full control over everything that's in the HTTP specification, including the thread model, connection life cycle, and chunked encoding.

And now let's dig inside Akka . Akka is a framework which provides an excellent abstraction on the top of Java concurrent API, and it comes with API in Java or Scala.

• It provides you a clear way to structure your application as a hierarchy of actors :
  • Actors communicate through message passing, using immutable message so that you have not to care about thread-safety
  • Actors messages are stored in message boxes, which can be durable
  • Actors are responsible for supervising their children
  • Actors can be run on one or more JVM and can communicate using a wide numbers of protocols
• It provides a lightweight abstraction for asynchronous processing , Future , which is easier to use then Java Futures.
• It provides other fancy stuff such as Event Bus, ZeroMQ adapter, Remoting support, Dataflow concurrency, Scheduler

---

Once you become familiar with the two frameworks, it turns out that what you need can easily be coded through them.

In fact, what you need is an http proxy coded in Netty, that upon a request receival sends immediately a message to an Akka Actor of type FSM (http://doc.akka.io/docs/akka/2.0.2/java/fsm.html) which using a durable mailbox (http://doc.akka.io/docs/akka/2.0.2/modules/durable-mailbox.html )

Answer 2

Here is a link to open-source library that was a Master Thesis of a student at Czech Technical University in Prague. It is very large and powerful library and mainly focuses on location. The good thing about it, though, is that it omitted the headers and other -ish that REST has.

It is the latest fork and hopefully it will give you at least inspiration for "own" solution.

Answer 3

how about those concurrent collections:

http://mcg.cs.tau.ac.il/projects/concurrent-data-structures

i hope that the license is ok .

Answer 4

You'll want to have a look to these to posts. (added at the end of the document)

Very basically an approach that works in a proficient way for me is to separate requests from the queue and the executor. Requests are executed as Runnables or Callables. Inherit from them to create different kind of requests to your API or service. Set them up there adding headers and or body prior to to executing them.

Enqueue those requests in a queue (choose which fits better for you - I'd say LinkedBlockingQueue will make the job) linked to an executor from within a bound service and calling them from your activity or any other scope. If you don't need to get responses and callbacks you can avoid using Guava for listening to futures or create your own callbacks.

I'll stay tuned. If you need more depth I can post some specific pieces of code. There's the source of a basic example in the first link though.

http://ugiagonzalez.com/2012/08/03/using-runnables-queues-and-executors-to-perform-tasks-in-background-threads-in-android/

http://ugiagonzalez.com/2012/07/02/theres-life-after-asynctasks-in-android/

Update:

You can create another queue for those requests that were impossible to execute. One approach that comes to my mind would be to add all your failed requests to the retry queue. The retry queue would be trying to re-run these tasks while the phone still thinks that there's any kind of internet connection available. In the request object you can set a max number of retrials and compare it to a currentRetry number increasing it in every retrial. Mmm this might be interesting. I'll definitely think about including that in my library.