Debug missing messages in akka

Question

I have the following architecture at the moment:

Load(Play app with basic interface for load tests) -> Gateway(Spray application with REST interface for incoming messages) -> Processor(akka app that works with MongoDB) -> MongoDB

Everything works fine as long as number of messages I am pushing through is low. However when I try to push 10000 events, that will eventully end up at MongoDB as documents, it stops at random places, for example on 742 message or 982 message and does nothing after.

What would be the best way to debug such situations? On the load side I am just pushing hard into the REST service:

for (i ← 0 until users) workerRouter ! Load(server, i)

and then in the workerRouter

WS.url(server + "/track/message").post(Json.toJson(newUser)).map { response =>
  println(response.body)
  true
}

On the spray side:

pathPrefix("track") {
  path("message") {
    post {
      entity(as[TrackObj]) { msg =>
        processors ! msg
        complete("")
      }
    }
  }
}

On the processor side it's just basically an insert into a collection. Any suggestions on where to start from?

Update: I tried to move the logic of creating messages to the Gatewat, did a cycle of 1 to 10000 and it works just fine. However if spray and play are involed in a pipeline it interrupts and random places. Any suggestions on how to debug in this case?

Answer 1

In a distributed and parallel environment it is next to impossible to create a system that work reliably. Whatever debugging method you use it will only allow you to find a few bugs that will happen during the debug session.

Once our team spent 3 months(!) while tuning an application for a robust 24/7 working. And still there were bugs. Then we applied a method of Model checking ( Spin ). Within a couple of weeks we implemented a model that allowed us to get a robust application. However, model checking requires a bit different way of thinking and it can be difficult to start.

Answer 2

I moved the load test app to Spray framework and now it works like a charm. So I suppose the problem was somewhere in the way that I used WS API in Play framework:

WS.url(server + "/track/message").post(Json.toJson(newUser)).map { response =>
  println(response.body)
  true
}

The problem is resovled but not solved, won't work on a solution based on Play.