使用文件写入操作将5M数据写入CSV文件

Question

I'm writing some data in a single column of a CSV file using file write operation. I am able to write values only in 1048576 rows. I have 5 million integer data values and I want it to be saved in a single CSV file. Below is my code

with open(path, 'w') as fp:
    for i in range(0,len(values)):
        fp.write(values[i] + '\n')
    fp.close()

• Is it possible to continue writing values after 1048576 rows to 3rd/4th column of the CSV file?? OR

• Is it possible to write values in a sequential way so that i can have all the values in a single file?

Answer 1

You can use itertools.izip_longest to "chunk" the values into "columns", then use the csv module to write those rows to the file. eg:

import csv
from itertools import izip_longest

N = 5 # adapt as needed
values = range(1, 23) # use real values here

with open(path, 'wb') as fout:
    csvout = csv.writer(fout)
    rows = izip_longest(*[iter(values)] * N, fillvalue='')
    csvout.writerows(rows)

This will give you the following output:

1,2,3,4,5
6,7,8,9,10
11,12,13,14,15
16,17,18,19,20
21,22,,,

You can also "transpose" the data so the data "runs the other way round", eg:

import csv
from itertools import izip_longest, izip

N = 5 # adapt as needed
values = range(1, 23) # use real values here

with open(path, 'wb') as fout:
    csvout = csv.writer(fout)
    rows = izip_longest(*[iter(values)] * N, fillvalue='')
    transposed = izip(*rows)
    csvout.writerows(transposed)

This will give you:

1,6,11,16,21
2,7,12,17,22
3,8,13,18,
4,9,14,19,
5,10,15,20,

Answer 2

As an alternative, you can use islice to give you the required number of columns per row as follows:

from itertools import islice
import csv

path = 'output.txt'
values = range(105)     # Create sample 'values' data
columns = 10
ivalues = iter(values)

with open(path, 'wb') as fp:
    csv_output = csv.writer(fp)
    for row in iter(lambda: list(islice(ivalues, columns)), []):
        csv_output.writerow(row)

Giving you the following:

0,1,2,3,4,5,6,7,8,9
10,11,12,13,14,15,16,17,18,19
20,21,22,23,24,25,26,27,28,29
30,31,32,33,34,35,36,37,38,39
40,41,42,43,44,45,46,47,48,49
50,51,52,53,54,55,56,57,58,59
60,61,62,63,64,65,66,67,68,69
70,71,72,73,74,75,76,77,78,79
80,81,82,83,84,85,86,87,88,89
90,91,92,93,94,95,96,97,98,99
100,101,102,103,104

Note, in your example, you should convert range to xrange to avoid Python creating a huge list of numbers to iterate on.