Pandas 保存到多个小 CSV 文件

Question

Im reading the CSV data from a stream (example cat /tmp/rawfile ). The below demo code works.

cmd = ('cat', '/tmp/csvfile')
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
csv = io.StringIO(process.stdout.read().decode())
data = pandas.read_csv(csv, index_col=0)
csv.close()

This CSV contains 10K rows. I want to read from Pandas and save 1k rows into a CSV file. Then the next 1k rows will go to the next CSV file. This is what I'm trying to achieve. Is this possible?

Some information about this:

I have some CSV files which really huge in size.(billions of rows) I used the Split command, but a few rows having \n - newline characters so while splitting based on a number of lines, it's moving the rest of the columns after the \n is going to the next row.

Example:

Row 1:
"col1" | "col2" | "This is
my first row"
Row 2:
"col1" | "col2" | "This is my second row"

In row 1 - These two lines belong to a specific column. But If I split based on lines, it'll split it into two different rows.

That's why I want to use Pandas to read the streaming data( stdin ) 100 rows per as a chunk and put it in a CSV file. Then read the next 100 rows and append the same CSV, because I want to put 1k rows per CSV.

Any suggestions or example code for this logic?

Update:

my intention is, put 1k rows in one CSV file. The reason why I'm read 100 rows per DF is, to save memory. Read 100 rows into DF and then flush it to a file, then next 100 --> repeat till 1000 rows (10 times), then repeat the whole process for next 1000 rows with a different csv file

Answer 1

Not sure if you're wanting this,try:

df=pd.DataFrame()
df=df.append(["x"]*1000)


# Loop
i=0

while i <len(df)-1:
   df.iloc[i:i+1000,:].to_csv("output_{0}_{1}.csv".format(i,i+1000),index=False)
   i+=1000