使用python根據特定字段重新格式化CSV

Question

http://example.com/item/all-atv-quad.html,David,"Punjab",+123456789123
http://example.com/item/70cc-2014.html,Qubee,"Capital",+987654321987
http://example.com/item/quad-bike-zenith.html,Zenith,"UP",+123456789123

我有這個test.csv，在這里我從某些站點刮了一些項目，但是“數字”字段具有冗余性。 因此，我需要以某種方式刪除具有與之前相同編號的行。 這只是示例文件，在實際文件中，某些數字重復了50多次以上。

import csv

with open('test.csv', newline='') as csvfile:
    csvreader = csv.reader(csvfile, delimiter=',')

    for column in csvreader:

        "Some logic here"

        if (column[3] == "+123456789123"):
            print (column[0])

            "or here"

我需要這樣重新格式化的csv：

http://example.com/item/all-atv-quad.html,David,"Punjab",+123456789123
http://example.com/item/70cc-2014.html,Qubee,"Capital",+987654321987

Answer 1

#!/usr/bin/env python
# -*- coding: utf-8 -*-


import pandas as pd


def direct():
    seen = set()
    with open("test.csv") as infile, open("formatted.csv", 'w') as outfile:
        for line in infile:
            parts = line.rstrip().split(',')
            number = parts[-1]
            if number not in seen:
                seen.add(number)
                outfile.write(line)


def using_pandas():
    """Alternatively, use Pandas"""
    df = pd.read_csv("test.csv", header=None)
    df = df.drop_duplicates(subset=[3])
    df.to_csv("formatted_pandas.csv", index=None, header=None)


def main():
    direct()
    using_pandas()


if __name__ == "__main__":
    main()

Answer 2

這將過濾出重復項：

seen = set()
for line in csvreader:
    if line[3] in seen:
        continue
    seen.add(line[3])
    # write line to output file

和csv讀寫邏輯：

with open('test.csv') as fobj_in, open('test_clean.csv', 'w') as fobj_out:
    csv_reader = csv.reader(fobj_in, delimiter=',')
    csv_writer = csv.writer(fobj_out, delimiter=',')
    seen = set()
    for line in csvreader:
        if line[3] in seen:
            continue
        seen.add(line[3])
        csv_writer.writerow(line)