在两个文件上设置操作差异

Question

I want to get diff of two flat/CSV files source and target which would have same schema.

let say,

source.txt:

EmpId|RegionId|Sales
001|R01|$10000
002|R02|$20000
003|R03|$30000

target.txt:

EmpId|RegionId|Sales
001|R01|$10000
002|R02|$10000
004|R04|$40000

Result should be:

EmpId1|RegionId1|Sales1|EmpId2|RegionId2|Sales2|Result_Status
001|R01|$10000|001|R01|$10000|matched
002|R02|$20000|002|R02|$10000|unmatched
003|R03|$30000|NULL|NULL|NULL|unmatched
NULL|NULL|NULL|004|R04|$40000|unmatched

any help would be appriciated!!

Edited:

Provided given 2 files are huge in size, this problem may look like simpler, but I am trying to find the best way of doing it, Performance is major criteria here, technology can be anything, even hadoop map reduce, I tried using Hive but it was bit slower.

Answer 1

Here is a map-reduce approach to solve it (in high level pseudo code):

map(source):
   for each line x|y|z:
     emitIntermediate(x,(1,y|z))
map(target):
   for each line x|y|z:
     emitIntermediate(x,(2,y|z))

//make sure each list is sorted/ sort it yourself 1 is before 2 if both exists.
reduce(x, list):
   if list.size() == 1:
      (idx,y|z) <- list.first() //this is the configuration of the element in the list
      if idx == 1:
            emit(x|y|z|NULL|NULL|NULL|unmatched)
      else:
            emit(NULL|NULL|NULL|x|y|z|unmatched)
   else:
       (1,y1|z1) <- list.first()
       (2,y2|z2) <- list.last()
       m = (y1|z1 matches y2|z2 ? "matched" : "unmatched")
       emit(x|y1|z2|x|y2|z2|m)

The idea is to split the data in the reduce part to the different IDs in the map phase, and let the reducers check if the region and sales matches.

Implementing it over a large cluster (and in a distributed file format) can improve performance significantly since the work is distributed across the cluster by the map-reduce framework.

You can use Hadoop as an implementing framework, for example.