Comparing two csv files based on multiple columns and save in separate file

Question

I have two files with same format where one has new updates and the other has older updates. There is no particular unique id column.

How can I extract the new updated lines only (with unix, PHP, AWK)?

Answer 1

You want to "byte" compare all lines against the other lines, so i would do:

$lines1 = file('file1.txt');
$lines2 = file('file2.txt');

$lookup = array();

foreach($lines1 as $line) {
  $key = crc32($line);
  if (!isset($lookup[$key])) $lookup[$key] = array();
  $lookup[$key][] = $line;
}

foreach($lines2 as $line) {
  $key = crc32($line);

  $found = false;
  if (isset($lookup[$key])) {
    foreach($lookup[$key] as $lookupLine) {
      if (strcmp($lookupLine, $line) == 0) {
        $found = true;
        break;
      }
    }
  }

  // check if not found
  if (!$found) {
    // output to file or do something
  }
}

Note that if the files are very large this will consume quite some memory and you need to use some other mechanism, but the idea stays the same