代码优化以将整数列表转换为 Java 中的对象列表

Question

I have around 50000 to 500000 employee Ids and I want those employee ids to convert into detail objects.

I have done something like this to achieve that:

private Set<Detail> setDetail(List<Integer> employees, Group group) {
  Set<Detail> details = employees.stream().parallel().map(id -> new Detail(id, group)).collect(Collectors.toSet());
  return details ;
}

But this is very slow and getting more slower with increase number of employee ids. How I can optimize this code? What are the optimization techniques/algorithms I can use to better optimize this.

Answer 1

You should try to avoid creating that many objects. No matter which algorithm you'll pick if your DB will keep growing at some point you won't fit it all in memory. Also the bottleneck is likely going to be getting data from DB (not creating objects).

So try to re-architect your app so that the data is pre-calculated and stored in DB at the time of performing the operation in question.

If after careful consideration you decide that you do need to work with that many objects a better option would be to keep working with primitives:

class EmployeesInGroup {
   private final int[] ids;
   private final Group group;
   ...

   Detail get(int idx) {
      return new Details(ids[idx], group);
   }

   int size() {
     return ids.length;
   }
}

Then you can iterate over this list and work with 1 object at a time w/o keeping a lot of them in memory:

EmployeesInGroup list = new EmployeesInGroup(ids, group);
for(int i = 0; i < list.size(); i++) {
  Detail d = list.get(i);
  ...
}

You can make it implement Iterable and use for-each loop.

Benchmarks

Approach that I listed above is at least 20x faster than creating an array of Detail objects. Working with streams and lists slows it all down even more. I didn't check with Integer but I'd predict it will slow everything down by another factor of 2 or something.

Benchmark                                        Mode  Cnt     Score    Error  Units
EmployeeConversionBenchmark.objectArray         thrpt   20   368.702 ±  3.483  ops/s
EmployeeConversionBenchmark.primitiveArray      thrpt   20  7595.080 ± 68.841  ops/s
EmployeeConversionBenchmark.streamsWithObjects  thrpt   20   197.923 ±  1.616  ops/s

Here's the code that I used:

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;

import java.util.Arrays;
import java.util.List;
import java.util.Random;

import static java.util.stream.Collectors.toList;

public class EmployeeConversionBenchmark {
    public static void main(String[] args) throws Exception {
        org.openjdk.jmh.Main.main(new String[]{EmployeeConversionBenchmark.class.getSimpleName()});
    }

    @Benchmark @Fork(value = 1, warmups = 0)
    public int primitiveArray(Data data) {
        EmployeesInGroup e = new EmployeesInGroup(data.ids, data.group);
        int sum = 0;
        for (int i = 0; i < e.size(); i++)
            sum += e.get(i).getId();
        return sum;
    }
    @Benchmark @Fork(value = 1, warmups = 0)
    public int objectArray(Data data) {
        EmployeesInGroup.Detail[] e = new EmployeesInGroup.Detail[data.ids.length];
        for (int i = 0; i < data.ids.length; i++)
            e[i] = new EmployeesInGroup.Detail(data.ids[i], data.group);

        int sum = 0;
        for (EmployeesInGroup.Detail detail : e)
            sum += detail.getId();
        return sum;
    }
    @Benchmark @Fork(value = 1, warmups = 0)
    public int streamsWithObjects(Data data) {
        List<EmployeesInGroup.Detail> e = Arrays.stream(data.ids).mapToObj(id -> new EmployeesInGroup.Detail(id, data.group)).collect(toList());
        int sum = 0;
        for (EmployeesInGroup.Detail detail : e)
            sum += detail.getId();
        return sum;
    }

    @State(Scope.Benchmark)
    public static class Data {
        private final int[] ids = new int[500_000];
        private final EmployeesInGroup.Group group = new EmployeesInGroup.Group();
        public Data() {
            for (int i = 0; i < ids.length; i++)
                ids[i] = new Random().nextInt();
        }
    }

    public static class EmployeesInGroup {
        private final int[] ids;
        private final Group group;

        public EmployeesInGroup(int[] ids, Group group) {
            this.ids = ids;
            this.group = group;
        }
        public Detail get(int idx) {
            return new Detail(ids[idx], group);
        }

        public int size() {
            return ids.length;
        }

        public static class Group {
        }

        public static class Detail {
            private final int id;
            private final Group group;

            public Detail(int id, Group group) {
                this.id = id;
                this.group = group;
            }
            public int getId() {
                return id;
            }
        }
    }
}

Answer 2

In general, a conversation about performance and optimization should include numbers, eg: the latency you are trying to achieve, the latency you actually observe, some details about your hardware, etc...

Your problem (if there is one) is not in the method you posted. On my laptop, the code below executes in ~250 ms. and even faster (~ 200 ms) if you remove parallel() :

class Detail {
    private Integer id;
    private String group;

    public Detail(Integer id, String group) {
        this.id = id;
        this.group = group;
    }
}

public class Main {
    public Set<Detail> setDetail(List<Integer> employees, String group) {
        return employees.stream().parallel().map(id -> new Detail(id, group)).collect(toSet());
    }

    public static void main(String[] args) {
        List<Integer> idsList = new Random().ints().boxed().limit(500_000).collect(toList());

        long start = System.currentTimeMillis();

        Set<Detail> details = new Main().setDetail(idsList, "group");

        long duration = (System.currentTimeMillis() - start);
        System.out.println("Done in " + duration + " ms. Size was " + details.size());
    }
}