处理与 tbb 顺序的大数据

Question

I'm working on c++ app to process large amounts of quote data eg. (MSFT, AMZN, etc) with tbb. And was wondering how I would structure it. I'm been looking at parallel_for and pipeline and concurrent_queue.

The process would basically parse the data, process it and output to file. Parsing and processing can be done in parallel, but output should be in order for each symbol.

Eg. Input:
    - Msg #1 - AMZN #1
    - Msg #2 - AMZN #2
    - Msg #3 - IBM #1
    - Msg #4 - AMZN #3
    - Msg #5 - CSCO #1
    - Msg $6 - IBM #2

I would like to use lock-free solution or minimum locking, but it seems like I have keep in concurrent_queue to keep the order.

Any ideas would be helpful

Thanks, David

Answer 1

If you use the pipeline pattern ( tbb::pipeline class or tbb::parallel_pipeline() function), you can use ordered filters to ensure the output will appear in exactly the same order as the input was received. And you will not need any locks in your code for ordering.

Answer 2

Does your quote data either have a timestamp or a sequence number
Otherwise add a sequence number from the producer thread and sort the data based on squence number after parsing it - the resorting can be done then either in a batch or just before the writing of the files.

Answer 3

You can create an output structure (hash or list) where a key is a position of the displayed element (1st, 2nd, ...) and the value is the data to be displayed. Then when all the elements are ready, you can output the structure in the desired order.

This way you don't care about which thread finishes first.