WSO2 CEP window time 1 day

Question

I am evaluating different possible solution for ATM Card transaction fraud detection with input load of around 50000 per second and response time few seconds.

WSO2 CEP looks better fit for overall solution, but stuck with problem of memory & performance as I am new to WSO2 CEP so please suggest if there is better way of doing below in CEP WSO2/CEP.

in order to detect fraud we have to capture data aggregation over a time period of 1 day which is causing either memory overflow or performance hit.

1) Below causing out of memory as CEP tries to keep all record in memory for whole day

from instream#window.time(1 day) 
select sum(amount) as totalAmt;

2) Below causing performance hit as it tries to load all records from table for doing some.

define table instream_table (....) from ( datasource,table,cache policy) ;

from instream#window.length(1) join instream_table
on instream.card_id==intable.card_id
select sum(instream_table.amount) as totalAmt;

worst thing I have noticed that CEP fires select * from instream_table instead of adding even where clause for card_id, Where I was expecting CEP to intelligent enough to fire select sum(amount) from instream_table where card_id=xxxxx

i have looked at documentation for window in WSO2 CEP but could not find any way to optimize this as it looks like WSO2 CEP tries to everything in memory.

let me know if there is any work around or better solution to achieve this. I have looked at other CEP engines like esper but seems every body is doing this in same manner.

Answer 1

It seems a known bug in event tables.. I have created a jira to track that issue in [1] and added a patch for that issue as well.. Will fix that issue in next release..

[1] https://wso2.org/jira/browse/CEP-866

Thanks..