需要帮助为业务活动监控（BAM）应用程序设计POC

Question

I have been researching tools and frameworks to implement a BAM application. The basic requirements will be:

1. Interface with different applications to get business status/activities. Initial feeds would be JMS provider, Webservices, FTP, and JMX.
2. Would be as near real time as possible.
3. Would need to process 20+ million messages a day with bursts of 2000/sec (messages as in XML over a JMS provider like activeMQ or WebsphereMQ).
4. Generate alerts when KPIs are breached (warning and critical levels)

Other things to consider is our small company has a Mulesoft EE license, a company application stack (osgi bundles with Tomcat, ActiveMQ, Drools, etc..)

So, the initial draft approach I was thinking of from research alone before I try a POC is something like:

Using Mule as an ESB to integrate with the various applications, and then using its SEDA service to process and stage events to the Drools engine to process rules associated with these events.

I am not quite sure if this process is the way these tools were designed to be used, or if there is a better more eloquent way to handle this. Also some other outstanding questions:

• How can aggregation of events and status (warnings, kpi breaches, etc) be displayed to the user. GWT integration? Should the events be put into an in-memory db to be queried and displayed?
• As far as the physical architecture, I was thinking of running the mule in clustered tomcat instances across two servers with the drools engines on the same servers? The database (for history storage) on its own servers.
• I am not limited to the tools above, I was also looking at Esper, Apache Camel
• Is this approach overkill? Can a more simple straight-forward webapp be used with RDBMS managing state? I figured the real-time and number of events requirements would prevent this.

ANY help would be appreciated to help me through an initial strategy here, or is anyone has implemented a solution with similar requirements and would like to share. Thank you!

Answer 1

You might want to consider using NoSQL database for storing event data because of

1. lots of data
2. Required speed of writing event per second
3. You need flexible model for capturing different attributes in event data specific to KPI definitions as KPIs are based on your app data model.

If you chose to go on this route then I would suggest using document oriented data store like Mongo DB because of

1. Requires efficient reads to calculates value for defined KPI.
2. Requires very rich data representation mainly hierarchical one for capturing event data.
3. You can employ secondary indexing mechanism like lucene to improve out of the box read performance provided by DB. These indexes can be specific to defined KPI based on App's data model.

Yes you need asynchronous event processing mechanism like MQ.

I had seen that IBM Websphere BAM server uses XML representation of its event data and they store them in relational DB like DB2. But I think NoSQL would be better choice.