为什么使用Java的流API收集器进行管道分组需要更多时间？

Question

I'm creating a Map of Map from a list using Stream API. List contains 10 million records. I have done this with single statement using two 'groupingBy' operations as below. The problem is that this one line statement is taking almost 1.5 minute to execute which became a bottleneck in execution in my performance critical application.

I have given the code I tried below using parallel stream API

Map<MyKey, Map<String, List<Person>>> personMap = personList.parallelStream()
    .collect(Collectors.groupingBy(
        person -> new MyKey(person.Id(), person.getPricePointId()),
        Collectors.groupingBy(Person::getWorkType)));

It is taking more than 1.5 minute to execute the above code which almost 75 % of my overall execution time. I do not find any other solution faster than this. So my question here is, Is this maximum possible throughput for this much volume of data ? or using downstream(multiple groupingBy) is not right option here ?, If not what should be the right way to reduce the execution time?

Answer 1

what youre are doing is a bad idea querying a 10 million records takes a lot of memory so, group your query into limit and start point and divide queries according to limit and start point and run every query in separate thread at last join the thread. it will be much more faster as well efficient for your use case