微服务顺序数据处理

Question

Suppose I am receiving a stream of unordered sequential data in time.

For example, input could be:

[
    {id:1, timestamp:1},
    {id:2, timestamp:1},
    {id:2, timestamp:2},
    {id:1, timestamp:2},
    {id:3, timestamp:1}
] 

Each entity is identified by 'id' field. There could be a large amount of entities and processing for each input could take some time. The problem is that I need to process each of those events in order it was received for each entity.

I was considering some solutions as to put messages to Kafka topic with partitions and receive parallelism? Or create local storage of received messages and marking each processed message for each entity after successful processing (on other machine or on the same in Thread pool)?

Questions: Is it is a good solution? How can I reach this functionality while scaling data consumers (having fixed number of services/ creating new instances)? Maybe there is a better way to solve such kind of problem?

Answer 1

"IF" the sequential data you mention just divided by id, 1 2 and 3, Then Would be the best you make 3 background services as an consumer, just need 1 partition for the case (you can decided this on your own)

Then make 3 topic based on the data ex : TOPIC 1 TOPIC 2 TOPIC 3

which mean you need to make 3 kind of consumer, each of consumer would be listen to only 1 topic

Then you would be spawn new process / Thread for every new stream data, It would work in parallel