How to compare the list/map of custom objects, field by field to create mismatch report for very big data set in generic way using Java 8 or more?

Question

I have been working on comparison of data between 2 different database sources in Java. I can't compare it in DB directly, due to some other challenges.

• I have 50 tables to compare.
• Table count will vary from 10k to 500k. (Need efficient algorithm)
• Each table column count and field names will also be different (of course)

I've written below code using for loop which is limitations like:

1. As data size can be huge for some tables, for loop solution is not efficient.
2. Number of columns will vary for each table, hence the logic I have written wont work for all, I need to repeat it for different tables. Lots of boilerplate code.
3. Suppose any new column is added to some table, comparison logic also needs to be updated

My requirement:

1. I want to write a efficient code for finding field by field mismatch report of provided list of custom objects.
2. The comparison code should be capable of comparing any type of custom object list. (Not sure how to do it)
3. Ability to create, table object POJO by referring to some property file which will contain the list of columns for all tables.

public void loadDummyTableObjects() {
        table1DataList =
                Arrays.asList(new TestTable1("1","1","One","Blue"),
                        new TestTable1("2","2","Two","Red"),
                        new TestTable1("3","3","Three","Black"),
                        new TestTable1("4","4","Four","Green"),
                        new TestTable1("5","5","Five","White"));

        table2DataList =
                Arrays.asList(new TestTable2("1","1","One","Blue"),
                        new TestTable2("2","2","Two","Red1"),
                        new TestTable2("3","3","Three","Black"),
                        new TestTable2("4","4","Four","Green"),
                        new TestTable2("5","5","Two","White"));
    }

   public void compareDataWithForLoop() {
        loadDummyTableObjects();
        List<MismatchReport> mismatchReport = new ArrayList<>();
        for (TestTable1 t1Row: table1DataList) {
            for (TestTable2 t2Row: table2DataList) {
                if (t1Row.getId().equals(t2Row.getId())) {
                    if (!(t1Row.getColumn1().equals(t2Row.getColumn1()))) {
                        MismatchReport result = getMismatchReport("Table1", "Column1", t1Row.getColumn1(), t2Row.getColumn1());
                        mismatchReport.add(result);
                    }
                    if (!(t1Row.getColumn2().equals(t2Row.getColumn2()))) {
                        MismatchReport result = getMismatchReport("Table1", "Column2", t1Row.getColumn2(), t2Row.getColumn2());
                        mismatchReport.add(result);
                    }
                    if (!(t1Row.getColumn3().equals(t2Row.getColumn3()))) {
                        MismatchReport result = getMismatchReport("Table1", "Column3", t1Row.getColumn3(), t2Row.getColumn3());
                        mismatchReport.add(result);
                    }
                }
            }
        }
        System.out.println(mismatchReport);
    }

    private static MismatchReport getMismatchReport(String tableNme, String Db1Table1Column1, String t1Row, String t2Row) {
        MismatchReport result = new MismatchReport();
        result.setTableNme(tableNme);
        result.setColumnNme(Db1Table1Column1);
        result.setDb1Value(t1Row);
        result.setDb2Value(t2Row);
        return result;
    }

    public static void main(String[] args) {
        DataComparatorService service = new DataComparatorService();
        service.compareDataWithForLoop();
    }

The output format for every table comparison should be same. The result should contain fields (TableName, ColumnName, Db1Value, Db2Value), to know the column for which difference has found and the mismatch values. The output of above code is:

[MismatchReport{tableNme='Table1', columnNme='Column3', db1Value='Red', db2Value='Red1'}, 
MismatchReport{tableNme='Table1', columnNme='Column2', db1Value='Five', db2Value='Two'}]

Any leads on how can above requirements be achieved will be very much helpful.

Answer 1

If I were you, instead of reinventing the wheel, I would work with a third party library such as JaVers.

JaVers Documentation

JaVers GitHub

Javers Maven

It is a powerful yet lightweight library. It can do much more but you can also use it as a mere object diff tool. As a starting point I have taken some of your example input to show how you can apply it to your use case.

import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;

import org.javers.core.Javers;
import org.javers.core.JaversBuilder;
import org.javers.core.diff.Diff;

import lombok.AllArgsConstructor;
import lombok.Getter;

public final class Example {

    public static void main(String[] args) {

        //Just copied your sample input but used only one custom class as the second is not really needed
        List<TestTable1> dataDB1 = Arrays.asList(new TestTable1("1","1","One","Blue"),
                      new TestTable1("2","2","Two","Red"),
                      new TestTable1("3","3","Three","Black"),
                      new TestTable1("4","4","Four","Green"),
                      new TestTable1("5","5","Five","White"));

        List<TestTable1> dataDB2 = Arrays.asList(new TestTable1("1","1","One","Blue"),
                      new TestTable1("2","2","Two","Red1"),
                      new TestTable1("3","3","Three","Black"),
                      new TestTable1("4","4","Four","Green"),
                      new TestTable1("5","5","Two","White"));

        //create a map from your input for a faster access of objects by id
        Map<String, TestTable1> db1Map = dataDB1.stream()
                                           .collect(Collectors.toMap(TestTable1::getId, Function.identity()));
        Map<String, TestTable1> db2Map = dataDB2.stream()
                                           .collect(Collectors.toMap(TestTable1::getId, Function.identity()));

        // do your comparison using JaVers
        Javers javers = JaversBuilder.javers().build();

        db1Map.keySet().forEach(key -> {
            Diff diff = javers.compare(db1Map.get(key), db2Map.get(key));
            if (diff.hasChanges()){
                System.out.println("Changes for id: " + key);
                System.out.println(diff.prettyPrint());
                System.out.println("********************************************************");
                System.out.println();
            }
        });
    }

    // a simple POJO for your data
    @AllArgsConstructor
    @Getter
    public static class TestTable1 {
        String id;
        String column1;
        String column2;
        String column3;
    }
}

Output:

Changes for id: 2
Diff:
* changes on com.mycompany.Example$TestTable1/ :
  - 'column3' changed: 'Red' -> 'Red1'

********************************************************

Changes for id: 5
Diff:
* changes on com.mycompany.Example$TestTable1/ :
  - 'column2' changed: 'Five' -> 'Two'

********************************************************

I just used prettyPrint to get a standard output but you can configure it to match your needs