Excel comparing two csv files and showing the difference

Question

I'm looking to compare two big sets of csv files and/or a csv file and a.txt file. I "think" the.txt file may need to be converted to a csv file just for simplicity sake but that may or may not be needed. I either want to use excel, c++, or python. I need to compare one "accepted" value list to a list that is measured and find the difference between them if there is one. Excel may be the easiest way to do this but python or c++ may work just as well. This is not homework so don't worry about that sort of thing. Code advice and/or templates is greatly appreciated. or links to websites

EDIT 1

I've read about Python's difflib or differ class but unfamiliar how to use it and may be more than I want.

EDIT 2

The Files both will have a series of columns(not with lines drawn between them or anything) and below those "named" columns there will be numbers. I need to compare the number in column 1 spot one in file one to column 1 spot one of file 2 and if there is a difference show the difference in another csv file

Answer 1

You can use ADO (ODBC/JET/OLEDB Text Driver) to treat 'decent'.txt/.csv/.tab/.flr files as tables in a SQL Database from every COM-enabled language. Then the comparisons could be done using the power of SQL (DISTINCT, GROUP, (LEFT) JOINS, ...).

Added with regard to your comment:

It's your problem and I don't want to push you where you don't want to go. But SQL is a good (the best?) tool, if you need to compare tabular data. As evidence the output of a script that spots the differences in two.txt files:

======= The .txt files to play with
------- file1.txt
"AC";"AM"
40000;-19083,00
40100;20000,00
40200;350004,00
40300;3498,99

------- file2.txt
"AC";"AM"
40000;-19083,00
40300;3498,99
40105;-234567,00
40200;350,00

======= Some diagnostic SQL
------- <NULL> indicates: In F1 but not in F2 (LEFT JOIN)
SELECT T1.AC, T1.AM, T2.AM FROM [file1.txt] AS T1 LEFT JOIN [file2.txt] AS T2 ON (T1.AC =
T2.AC)
------- Result
AC      File1   File2
40000   -19083  -19083
40100   20000   <NULL>
40200   350004  350
40300   3498,99 3498,99

------- <NULL> indicates: Not in the other file (LEFT JOIN, UNION)
SELECT T1.AC, T1.AM, T2.AM FROM [file1.txt] AS T1 LEFT JOIN [file2.txt] AS T2 ON (T1.AC =
T2.AC) UNION SELECT T2.AC, T1.AM, T2.AM FROM [file2.txt] AS T2 LEFT JOIN [file1.txt] AS T1
 ON (T1.AC = T2.AC)
------- Result
AC      File1   File2
40000   -19083  -19083
40100   20000   <NULL>
40105   <NULL>  -234567
40200   350004  350
40300   3498,99 3498,99

------- the problems: missing, different values
SELECT T1.AC, T1.AM, T2.AM FROM [file1.txt] AS T1 LEFT JOIN [file2.txt] AS T2 ON (T1.AC =
T2.AC) WHERE T2.AM IS NULL OR T1.AM <> T2.AM UNION SELECT T2.AC, T1.AM, T2.AM FROM [file2.
txt] AS T2 LEFT JOIN [file1.txt] AS T1 ON (T1.AC = T2.AC) WHERE T1.AM IS NULL OR T1.AM <>
T2.AM
------- Result
AC      File1   File2
40100   20000   <NULL>
40105   <NULL>  -234567
40200   350004  350

Further additions:

This article deals with ADO and text files; look for a file adoNNN.chm (NNN=Version number, eg 210) on your computer; this is a good book about ADO.

You can use Access or OpenOffice Base to experiment with SQL statements applied to a linked/referenced (not imported.) text database.

A script/program will be easy after you mastered the initial hurdle: connecting to the the database, ie to a folder containing the files and a schema.ini file to define the structure of the files=tables.

The output above was generated by:

  Const adClipString = 2

  Dim oFS  : Set oFS = CreateObject( "Scripting.FileSystemObject" )
  Dim sDir : sDir    = oFS.GetAbsolutePathName( ".\txt" )
  Dim oDB  : Set oDb = CreateObject( "ADODB.Connection" )
  oDB.Open "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & sDir & ";Extended Properties=""text"""
  Dim sSQL
  Dim sFiNa
  WScript.Echo "=======", "The .txt files to play with"
  For Each sFiNa In Array( "file1.txt", "file2.txt"  )
      WScript.Echo "-------", sFiNa
      WScript.Echo oFS.OpenTextFile( "txt\" & sFiNa ).ReadAll()
  Next

  WScript.Echo "=======", "Some diagnostic SQL"
  Dim aSQL
  For Each aSQL In Array( _
       Array(   "<NULL> indicates: In F1 but not in F2 (LEFT JOIN)" _
              , Join( Array( _
                     "SELECT T1.AC, T1.AM, T2.AM FROM" _
                   , "[file1.txt] AS T1" _
                   , "LEFT JOIN [file2.txt] AS T2 ON (T1.AC = T2.AC)" _
                ), " " ) ) _
     , Array(   "<NULL> indicates: Not in the other file (LEFT JOIN, UNION)" _
              , Join( Array( _
                     "SELECT T1.AC, T1.AM, T2.AM FROM" _
                   , "[file1.txt] AS T1" _
                   , "LEFT JOIN [file2.txt] AS T2 ON (T1.AC = T2.AC)" _
                   , "UNION" _
                   , "SELECT T2.AC, T1.AM, T2.AM FROM" _
                   , "[file2.txt] AS T2" _
                   , "LEFT JOIN [file1.txt] AS T1 ON (T1.AC = T2.AC)" _
                ), " " ) ) _
     , Array(   "the problems: missing, different value" _
              , Join( Array( _
                     "SELECT T1.AC, T1.AM, T2.AM FROM" _
                   , "[file1.txt] AS T1" _
                   , "LEFT JOIN [file2.txt] AS T2 ON (T1.AC = T2.AC)" _
                   , "WHERE T2.AM IS NULL OR T1.AM <> T2.AM" _
                   , "UNION" _
                   , "SELECT T2.AC, T1.AM, T2.AM FROM" _
                   , "[file2.txt] AS T2" _
                   , "LEFT JOIN [file1.txt] AS T1 ON (T1.AC = T2.AC)" _
                   , "WHERE T1.AM IS NULL OR T1.AM <> T2.AM" _
                ), " " ) ) _
     )
     sSQL = aSQL( 1 )
     WScript.Echo "-------", aSQL( 0 )
     WScript.Echo sSQL
     Dim oRS : Set oRS = oDB.Execute( sSQL )
     WScript.Echo "------- Result"
     WScript.Echo Join( Array( "AC", "File1", "File2" ), vbTab )
     WScript.Echo oRS.GetString( adClipString, , vbTab, vbCrLf, "<NULL>" )
  Next
  oDB.Close

If you delete/ignore the fat (create SQL statements, diagnostics output), it boils down to 6 lines

  Dim oDB  : Set oDb = CreateObject( "ADODB.Connection" )
  oDB.Open "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & sDir & ";Extended Properties=""text"""
  sSQL = "..."
  Dim oRS : Set oRS = oDB.Execute( sSQL )
  WScript.Echo oRS.GetString( adClipString, , vbTab, vbCrLf, "<NULL>" )
  oDB.Close

which can be 'ported' easily to every COM-enabled language, because the ADO objects do all the heavy lifting. The.GetString method comes handy, when you want to save a resultset: just twiddle the separator/delimiter/Null arguments and dump it to file

  oFS.CreateTextFile( ... ).WriteLine oRS.GetString( _
    adClipString, , ",", vbCrLf, ""
  )

(don't forget to add a definition for that table to your schema.ini). Of course you also can use a "SELECT/INSERT INTO", but such statements may not be easy to get right/passed the ADO Text Driver's parser.

Addition wrt Computations:

Start with a 5 x 2 master/approved file containing:

Num0    Num1    Num2    Num3    Num4
7,6     6,1     3,8     0,9     8,9
0,9     9,4     4,7     8,8     9,9

transform it to expected.txt

Num0    Num1    Num2    Num3    Num4    Spot
7,6     6,1     3,8     0,9     8,9     1
0,9     9,4     4,7     8,8     9,9     2

by appending the Spot column so it conforms to

[expected.txt]
ColNameHeader=True
CharacterSet=1252
Format=Delimited(;)
Col1=Num0 Float
Col2=Num1 Float
Col3=Num2 Float
Col4=Num3 Float
Col5=Num4 Float
Col6=Spot Integer

in your schema.ini file. Similarly, transform a measure file like:

Num0    Num1    Num2    Num3    Num4
7,1     1,1     3,8     0,9     8,9
0,9     9,4     4,7     8,8     9,9

to measured.txt

Num0    Num1    Num2    Num3    Num4    Spot
7,1     1,1     3,8     0,9     8,9     1
0,9     9,4     4,7     8,8     9,9     2

Apply

  sSQL = Join( Array( _
         "SELECT E.Num0 - M.Num0 AS Dif0" _
      ,       ", E.Num1 - M.Num1 AS Dif1" _
      ,       ", E.Num2 - M.Num2 AS Dif2" _
      ,       ", E.Num3 - M.Num3 AS Dif3" _
      ,       ", E.Num4 - M.Num4 AS Dif4" _
      ,       ", E.Spot          AS Spot" _
      ,  "FROM [expected.txt] AS E" _
      ,  "INNER JOIN [measured.txt] AS M" _
      ,  "ON E.Spot = M.Spot" _
  ), " " )

Write the resultset to differences.txt

aFNames = Array( "Num0", ... "Spot" ) oFS.CreateTextFile( sFSpec ).Write _ Join( aFNames, sFSep ) & sRSep & oRS.GetString( adClipString, , sFSep, sRSep, "" )

and you get:

Num0    Num1    Num2    Num3    Num4    Spot
0,5     5       0       0       0       1
0       0       0       0       0       2

Answer 2

You don't need to code, you can make separators the same in both files (spaces or commas) using replace function in text editor and compare it using graphical diff tool from TortoiseSVN: http://tortoisesvn.net/