Python数据框块列索引不正确

Question

I am learning DataFrame Chunking. My pseudocode is simple:

1. Break down the SOURCE_FILE into a number of chunks
2. Load a chunk (with a loop)
3. Add a column with a predicted label & another with confidence
4. Write the chunk to the drive
5. Continue the loop

The first chunk was saved as it was supposed to. The new columns in the rest of the chunks have incorrect row indexes. I cannot figure out why this is happening. Will appreciate all help.

Also, is my chunking pseudocode correct? I am a little confused if this is the right way.

# create chunks
for chunk in pd.read_csv(SOURCE_FILE, chunksize = CHUNK_SIZE):
    print('BATCH:', BATCH_NUMBER)
    
    # machine translate
    for row_index, text in enumerate(chunk.title):
        print('Text:', text)
        print('Row Index:', row_index)
        (label, confidence) = MODEL.predict(text)
        label = label[0]
        confidence = confidence[0]
        chunk.loc[row_index, 'Language'] = label[9:]
        chunk.loc[row_index, 'Confidence'] = confidence

    chunk.to_csv('Chunks/chunk' + str(BATCH_NUMBER) + '.csv' , index = False)
    BATCH_NUMBER += 1

You can see an image of the incorrect row indexing here

Answer 1

This is a little late, but I wanted to answer this in case anyone has the same problem in the future. You are attempting to concatenate two new columns to an already existing chunk with two columns. Pandas concatenates columns side-by-side using the index of the columns, and the index of your first chunk matches the index of your two new columns, but the rest of the chunks have larger indexes, and therefore they aren't joined properly. Reset the indexes of the rest of your chunks using

chunk.index = range(len(chunk))

And it should work.