如何根据 Python 中的特定变量（不是逐行）比较两个 csv 文件？

Question

I have two csv files I converted from Json file (copy the text in EXCEL and convert to csv), the format is a bit messy, I want to compare each whole line according to the ID number, but the problem is ID number is in different columns for each row, and I want to print the difference between two lines which have the same ID number.

Here is the data sample (i can't rename each column because each column has different variable values):

CSV_01:

age 10   height 150   ID  1001     sex F
age 10   height 150   ID  1001     sex M
ID 1001  height 150   age  12      sex M
age 10   ID  2002     height 151   sex F
age 10   height 150   ID  2002     sex M

CSV_02:

age 10   height 150   ID  2002     sex F
age 10   height 150   ID  1001     sex M
ID 1001  height 150   age  12      sex M
age 10   ID  1001     height 151   sex F
age 10   height 150   ID  2002     sex M

I have almost 1000 rows&500ish columns (and for each row it also contains duplicated same ID) Something like: age 10 height 150 ID 1001 sex M... ID 1001...

But I assume it doesn't matter, but the variables have different orders, which means ultimately I want to compare the first 3 rows in CVS_01 with the 2nd, 3rd,4th row in CSV_02 (because they have the same ID), but this is just an example so it should be different row numbers in my large data set.

Here's what I've tried after importing csv files in Python:

resultBool01 = (CSV_01 != CSV_02).stack()  # Create Frame of comparison booleans
resultdiff01 = pd.concat([CSV_01.stack()[resultBool01], CSV_02.stack()[resultBool01]], 
axis=1)
resultdiff01.columns=["output_01", "output_02"]

This gave me the difference between each row (ie: the first row between two files), but this is not what I want, because in the first row they have different ID. I'm stuck for a few days already, not sure if this is the right direction, but it could be more difficult if I compare json or txt files. Can someone help me? Many thanks.

Answer 1

Have you tried converting your data into a dictionary?

While the csv files are a bit messy, at least they have a clearly defined structure and each fieldname is present before its referenced value.

Whitespaces and other special characters notwithstanding, you could parse each csv line for line first, save each line (or entry) as a dictionary of unique datapoints and append it to a list of dictionaries. Then, you can either operate on it directly, or you can export a properly sorted and aligned csv file for later use.

(I cannot comment yet, so I hope this suffices, otherwise I am happy to help with the actual code, too)

Addendum:

The code might not be perfect for your specific file, but it can serve as a blueprint to develop yours.

Basically, each row as represented in csv (really tsv) format is:

fieldname   value   fieldname2   value2   fieldname3   value3

and this code will read it in and save the value2 as the value of the key "fieldname2" in a new dictionary, which we then save in a list and later return from the function

def filereader(filename):
        _out = []
        with open(filename, 'r',  newline='', encoding='?????') as csvfile:
                csvfile = csv.reader(csvfile, delimiter='\t')
                for row in csvfile:
                        rowDict = {}
                        rowDict[row[0]] = row[1]
                        rowDict[row[2]] = row[3]
                        rowDict[row[4]] = row[5]
                        _out.append(rowDict)
        return _out