Event Data Aggregation Using Kafka Streaming

Question

public class UserEvent {
    int userId;
    long loginTime;
    long jobId;
    long jobAttachTime;
    long jobdetachTime;
    long workTime;
    long logoutTime;
    long activeTime;
    EventType eventType;
}

I have an application where events are sent on a Kafka topic based on user actions like login, job attach, job detach and logout. Each event has some information in a UserEvent object along with userId and eventType , for example a Login Event has loginTime ; a Job Attach event has the properties jobId , jobAttachTime . Similarly a Logout event has the property logoutTime . My requirement is to aggregate information from all these events into one object after receiving the Logout event for each user. So that after the logout event, the UserEvent object will have loginTime , logoutTime , calculated workTime , activeTime etc. How could this be achieved using Kafka KStreams and/or KTables ?

Answer 1

In order to aggregate the UserEvents, you need a key(example : session id) to filter the sessions which is common across all the events. Let's say you have sessionID attached to each user event , which is unique for each session but same for all user events occured during that session.

It can be achieved using GroupBy().aggregate() in following way : ( considering you have session Id equivalent attribute which can be unique to use as a key )

    // Let's say there is a  sessionID
KTable<String, UserEvent> userEventSummary = userEvents
                                  .groupBy(event -> event.get("sessionId"))
                                  .aggregate((userEventSummary,userEvent)->{
                                        userEventSummary = userEvent;
                                        if(new!= null){
                                            String loginEvent = new.get("eventType").get("eventName");
                                            if(loginEvent.equals("login")){
                                                userEventSummary.setLoginTime(new.getLoginTime());
                                            }
                                            if(loginEvent.equals("logOut")){
                                                long workTime = Math.abs(userEvent.getLogOutTime()-userEventSummary.getLoginTime());
                                                userEventSummary.setWorkTime(workTime);
                                                userEventSummary.setActiveTime(workTime);
                                            }
                                        }
                                        return userEventSummary;
                                   });

// if default value for logoutTime is 0, filter the user events which don't have logout time yet
KTable<String, UserEvent>  loggedOutEventSummary = userEventSummary.filter(event-> event.getLogOutTime()!= 0);

It will return the aggregated state for each user action filtered by user events which have Logout events.

Answer 2

final KStream<Integer, UserEvent> kStream = builder.stream("test-topic");
kStream.groupByKey().aggregate(() -> new UserEvent(),
                (Integer userId, UserEvent userEvent, UserEvent userEventSummary) -> {
                    if (userEvent.getEventType().equals(EventType.LOGIN)) {
                        // Event Processing logic for Login event
                    } else if (userEvent.getEventType().equals(EventType.LOGOUT)) {
                        // Event Processing logic for Logout event
                    }
                    return userEventSummary;
                });