Reconcillation - any java based library?

Question

I am coding a java based component to do some reconcilliation of about a million data items (hence performance is of paramount importance).

The two sources of data are 1. A table in a database 2. A flat file.

There are a couple of columns in the database, that should match up to the a couple of columns in the file. The data are not sorted or anything. So each data item (lets say it is a name and corresponding phone number) should be same in the database and in the flat file. Eg if one of the rows in database says for "john" the number is "220 343 123" then the row in the flat file for "john" must have number as "220 343 123". If not, I report a break and move on.

Of course I could pick up each row (from database) and try to match it to some row in the file (horrible persformance). Or I could, sort the table and the file (somehow) before I do that (still clunky) Or ... something else

The question to the forum is, do you know of any library / tool (in java world) that could be leveraged for this?

Thanks.

Answer 1

You could read the flat file and put the data in some kind of Map structure. Next, you iterate through the database lines and compare with the values in the Map .

This way you only iterate both sources of data once...