改進python csv處理循環

Question

我有一個csv文件，我需要從多個列中導出數據。 第4列包含行名。 只要此列中的值發生更改，我導出的數據就需要寫入新的單獨文件。 以下代碼有效，但速度很慢。 關於如何改進這個的任何提示？

增加：數據樣本：

Altitude,Date,Db,Depth,Linename,Qmag,TF,TF_HP,X,X_ob,X_org,Y,Y_ob,Y_org
10.87,10/2/2015,148,21.8342,10,1316,48831.936,0.060026123,506479.5515,506479.46,506479.46,5726744.3,5726743.73,5726743.73
10.84,10/2/2015,148,21.8342,11,1316,48831.969,0.092713686,506479.7927,506479.77,506479.77,5726744.443,5726744.2,5726744.2
10.85,10/2/2015,148,21.8669,11,1313,48832.014,0.137400275,506479.9672,506479.77,506479.77,5726744.741,5726744.2,5726744.2
10.82,10/2/2015,148,21.8342,12,1311,48831.969,0.092093953,506480.1677,506479.92,506479.92,5726744.945,5726744.44,5726744.44
10.83,10/2/2015,148,21.8669,12,1309,48831.969,0.091807708,506480.326,506480.08,506480.08,5726745.195,5726744.68,5726744.68

Python代碼：

import glob,csv,os, itertools

list_of_files = glob.glob('C:/test/*.csv')

directory = 'C:/test/conv/'
if not os.path.exists(directory):
     os.makedirs(directory)

for filename in list_of_files:

    with open(filename,"r") as source:
        header_line = next(source)
        rdr= csv.reader( source, delimiter = ',',lineterminator='\n' )

        x=0

        for row in itertools.islice(rdr,0,None):
            itemRow4 = row[4]
            outfileName=directory+itemRow4+'.csv'

            with open(outfileName,"a") as result:

                wtr=csv.writer(result, lineterminator='\n')

                if x == 0:
                    previousitemRow4 = row[4]
                    x = x+1                 

                if previousitemRow4 == itemRow4: 
                    wtr.writerow((row[8],row[11],row[6],row[0]))
                    previousitemRow4 = itemRow4
                if previousitemRow4 != itemRow4:
                    wtr.writerow((row[8],row[11],row[6],row[0]))
                    print 'next line'
                    previousitemRow4 = itemRow4

Answer 1

使用標准的Unix shell工具cut ， sort ， uniq和grep ：

$ cut -d, -f5 < in.csv | sort | uniq | while read lineno
do
  grep ",${lineno}," in.csv > out-${lineno}.csv
done
$ ls out-*.csv
out-10.csv  out-11.csv  out-12.csv out-Linename.csv

也許grep表達式不夠復雜，因為lineno可能出現在其他列而不是第5列。 在這種情況下，可以使用簡單的正則表達式使grep僅在第5列中找到lineno。

Answer 2

根據Eumiro的建議，我提出了這個解決方案。 我嘗試使用列表的字典，但無法使其工作。 以下解決方案有效且速度非常快。 謝謝大家的幫助！

import glob,csv,os, itertools

list_of_files = glob.glob('C:/test/*.csv')
print 'By the Power of the Python, Split these here files!'  

directory = 'C:/test/conv/'
if not os.path.exists(directory):
    os.makedirs(directory)

for filename in list_of_files:
    storage = []
    specialStorage = []
    with open(filename,"r") as source:
        header_line = next(source)
        rdr= csv.reader( source, delimiter = ',',lineterminator='\n' )
        x=0
        resetValue = 0

        for row in itertools.islice(rdr,0,None):
            itemRow4 = row[4]

            if x == 0:
                previousitemRow4 = row[4]
                x = x+1

            outfileName=directory+previousitemRow4+'.csv'        

            if previousitemRow4 == itemRow4:                 

                storage.append((row[8],row[11],row[6],row[0]))
                previousitemRow4 = itemRow4

            if previousitemRow4 != itemRow4:

                with open(outfileName,"a") as result:

                    wtr=csv.writer(result, lineterminator='\n')
                    previousitemRow4 = itemRow4

                    if len(specialStorage) !=0:
                        wtr.writerow(specialStorage)
                    wtr.writerows(storage)
                    storage = []

                specialStorage=(row[8],row[11],row[6],row[0])

        else:
            with open(outfileName,"a") as result:

                    wtr=csv.writer(result, lineterminator='\n')
                    previousitemRow4 = itemRow4
                    wtr.writerow(specialStorage)
                    wtr.writerows(storage)
                    storage = []

            print 'end of file reached'