How to loop through each excel file beginning & ending with specific rows in a folder using Python

Question

I want to loop through each excel file in a folder, all files beginning with 5th row & ending with specific rows. I know how to loop through all files in a folder, the code is shown as below.

path = '/Users/XXXXX/Desktop/learning_append/all files'
files = os.listdir(path)

files_xlsx = [f for f in files if f[-4:] == 'xlsx']
files_xlsx

for f in files_xlsx:
    data = pd.read_excel(f, 'file1')
    df = df.append(data)    
df

I also know how to code for beginning with 5th row & ending with specific rows as shown below.

ending_index = df[df['Column1']== "Amendment:"].index.values
df.iloc[4:list(ending_index)[0]-1,:]

I think I need to define a function to the code above in order to do the same thing for each file in the folder. However, I don't know how to define & combine these codes to achieve what I want, folks please help me out here.

Answer 1

You are basically there if you put your filtering logic within the for loop:

1. Using skiprows is useful to remove individual rows at the top that you know. You might want to do this if you have column headers that you want to keep. If you know the number of rows you want to return you can also use nrows=...

2. Are you are trying to use panda functions on a List of panda dataFrames or a pandas df. You can do the filtering you need within the loop then append to the List (or dataframe).

3. df = df.append() actually returns 'None' if using a list. You may just want df.append()

eg

for f in files_xlsx:
    data = pd.read_excel(f, 'file1', skiprows=[0,1,2,3,4])
    
    ending_index = data[data['Column1']== "Amendment:"].index.values[0]
    data = data.iloc[:ending_index,:] 

    df.append(data)    
pd.concat(df) #If using a list