Accumulating count of answers using Java lambdas

Question

I have list of form data entries, each DataEntry contains data field which is of type Map<String, Object> . The String key is the dynamic field name (survey question), and value is the choice answered for that particular question.

Out of the list, how can I get count of unique answers for each field name (question), something like this I assume? Map<String, LinkedTreeMap<String, Integer> , where the outer map key is the fieldXXXX, its value map key (string) is the unique answer, and key Integer the count of that answer

For testing purpose, from database they come out as Strings which I map to DTOs where the mapper maps to proper json map:

DataEntry entry1 = new DataEntry();
entry1.setData("{field9294: '4', field9296: '3', field9319: '5', field9321: '5', field9323: '3', field9325: '3', field9327: '5', field9329: '7'}");
DataEntry entry2 = new DataEntry();
entry2.setData("{field9294: '3', field9296: '2', field9319: '3', field9321: '3', field9323: '5', field9325: '2', field9327: '4', field9329: '4'}");
DataEntry entry3 = new DataEntry();
entry3.setData("{field9294: '5', field9296: '5', field9319: '4', field9321: '4', field9323: '3', field9325: '3', field9327: '4', field9329: '8'}");

List<DataEntry> dataEntries = Arrays.asList(entry1, entry2, entry3);

List<FormDataDTO> dtos = dataEntries.stream().map(mapper::dataEntryToDto).collect(Collectors.toList());

So the list of dtos looks like this:

End goal

Let's take first field field9294 , in 3 data entries there are 3 unique answers given: 4, 3, 5. Here all should have count of 1. Now field9327 has answers 5, 4, 4. Here we count 5 once, 4 twice.

General idea would be to plot the data for each question separately so I can draw a chart and decorate the result as percentage for example.

As I see then getting Map<String, LinkedTreeMap<String, Integer> would be enough to achieve that, but is there any efficient way using fancy lambda tricks as I haven't been able to figure out myself. For result I expect something like this:

Map
 key: "field9294"
 values: "4" -> 1
         "3" -> 1
         "5" -> 1
 key: "field9327"
 values: "5" -> 1
         "4" -> 2

etc..

Thanks in advance!

Edit: All pass, thanks for the solution!

assertEquals("[3=1, 4=1, 5=1]", answerCountsByField.get("field9294").entrySet().toString());
assertEquals("[2=1, 3=1, 5=1]", answerCountsByField.get("field9296").entrySet().toString());
assertEquals("[3=1, 4=1, 5=1]", answerCountsByField.get("field9319").entrySet().toString());
assertEquals("[3=1, 4=1, 5=1]", answerCountsByField.get("field9321").entrySet().toString());
assertEquals("[3=2, 5=1]", answerCountsByField.get("field9323").entrySet().toString());
assertEquals("[2=1, 3=2]", answerCountsByField.get("field9325").entrySet().toString());
assertEquals("[4=2, 5=1]", answerCountsByField.get("field9327").entrySet().toString());
assertEquals("[4=1, 7=1, 8=1]", answerCountsByField.get("field9329").entrySet().toString());

Edit2: Looking for solution also to this structure . For results I only care about true answers, false are redundant for plotting as this structure maps to checkbox list

{"field6696":{"1":true,"2":true},"field7994":{"1":true,"2":false,"3":false,"4":true}}
{"field6696":{"1":false,"2":true},"field7994":{"1":false,"2":true,"3":true}}
{"field6696":{"1":false,"2":true},"field7994":{"1":false,"2":true,"3":false,"4":true}}
{"field6696":{"1":false,"2":true,"3":true},"field7994":{"1":true,"2":true,"3":false}}
{"field6696":{"1":false,"2":true},"field7994":{"1":true,"2":true,"3":true,"4":true}}

Answer 1

You should start with a Stream, so I would avoid collecting the FormDataDTO objects into a List. Change this:

List<FormDataDTO> dtos = dataEntries.stream().map(mapper::dataEntryToDto).collect(Collectors.toList());

to just this:

Stream<FormDataDTO> dtos = dataEntries.stream().map(mapper::dataEntryToDto);

Then you can collect them using a groupingBy call that itself uses another Collector to create Map values:

Map<String, Map<String, Long>> answerCountsByField =
    dtos.flatMap(dto -> dto.getData().entrySet().stream()).collect(
        Collectors.groupingBy(e -> e.getKey(),
            Collectors.groupingBy(e -> e.getValue(),
                Collectors.counting())));

If you want the counts to be Integers rather than Longs, you can use collectingAndThen to transform each Long value:

Map<String, Map<String, Integer>> answerCountsByField =
    dtos.flatMap(dto -> dto.getData().entrySet().stream()).collect(
        Collectors.groupingBy(e -> e.getKey(),
            Collectors.groupingBy(e -> e.getValue(),
                Collectors.collectingAndThen(
                        Collectors.counting(), Long::intValue))));

Answer 2

I wrote something this for now and that structure actually works for both cases as the end result can be exactly the same, only difference is the parsing as I have to go deeper by one map. I tried the same with the functions, then I had to check the type where is entry.getValue() as in one case it's String for other case it's another Map, but it really looked bad.

I put the grouping logic into the test to post here, data includes both types of structured data, single value and objects.

Maybe could suggest any improvements here?

  @Test
  public void multiValueLists_answerCountsByField() throws Exception {

    List<DataEntry> entries = new ArrayList<DataEntry>() {
      {
        add(new DataEntry("{field1000: {1: true, 2: true, 3: true}, field2000: {1: true, 2: true}, field3000: '1', field4000: '1', field5000: '1'}"));
        add(new DataEntry("{field1000: {1: true, 2: true, 3: false}, field2000: {1: true, 2: false}, field3000: '1', field4000: '2', field5000: '2'}"));
        add(new DataEntry("{field1000: {1: false, 2: true, 3: true}, field2000: {1: true}, field3000: '1', field4000: '2', field5000: '3'}"));
      }
    };

    Stream<FormDataDTO> dtoStream = entries.stream().map(mapper::dataEntryToDto);

    Map<String, Map<String, AtomicInteger>> answers = new LinkedTreeMap<>();

    dtoStream.forEach(dto -> dto.getData().entrySet()
      .forEach(field -> {
        answers.putIfAbsent(field.getKey(), new LinkedTreeMap<>());
        Map<String, AtomicInteger> values = answers.get(field.getKey());

        if (field.getValue() instanceof Map)
          ((Map<String, Boolean>) field.getValue()).entrySet().stream()
            .filter(value -> Boolean.TRUE.equals(value.getValue()))
            .forEach(value -> {
              values.putIfAbsent(value.getKey(), new AtomicInteger());
              values.get(value.getKey()).incrementAndGet();
            });
        else {
          values.putIfAbsent(field.getValue().toString(), new AtomicInteger());
          values.get(field.getValue().toString()).incrementAndGet();
        }
      }));

    assertThat(field(answers, "field1000"), is("[1=2, 2=3, 3=2]"));
    assertThat(field(answers, "field2000"), is("[1=3, 2=1]"));
    assertThat(field(answers, "field3000"), is("[1=3]"));
    assertThat(field(answers, "field4000"), is("[1=1, 2=2]"));
    assertThat(field(answers, "field5000"), is("[1=1, 2=1, 3=1]"));
  }