how to fill up data with given rows from another file in python

Question

Supposed I have 100 txt files, and every file has 20000 records, I would like to let every file have 25000 records, how to fill up data with another file to get every file with 25000 records?

Answer 1

when they are all in one directory, use this:

import os
import pandas as pd

path = "path/to/directory/"
dfs = []  # list of dataframes

for file in os.listdir(path):
    if file.endswith(".txt"):
        # edit with you separator of choice
        dfs.append(pd.read_csv(file, sep=" ")

# edit with your axis of choice
# ignore axis is important so you don't have multiple indices
full_df = pd.concat(dfs, axis=0, ignore_index=True)

l = len(full_df)
n_dfs = l // 25000 + 1  # new number of dfs
for i in range(ndfs):
    if i < (n_dfs - 1):
        new_df = full_df[i * 25000: (i+1) * 25000]
    else:
        new_df = full_df[i * 25000:]
    new_df.to_csv("path/to/new_df/file.txt", header=None, index=None, sep=' ', mode='a')

this should do.