如何在循环中将许多文件附加到数据框中

Question

I am trying to extract data from many docs files and append them into a dataframe.

The code I had written works great when it comes to a single file, but I cant seem to append into the dataframe for more files.

import re
import docx2txt
import pandas as pd
import glob

df2=pd.DataFrame()
appennded_data=[]

for file in glob.glob("*.docx"):
    text = docx2txt.process(file)
    a1=text.split()
    d2=a1[37]
    doc2=re.findall("HB0....",text)
    units2=re.findall("00[0-9]...",text) 
    df2['Units']=units2
    df2['Doc']=doc2[0]
    df2['Date']=d2
df2

This gives an error "Length of values does not match length of index"

Expected output-

From docx1: (Which I get)

Units |  Doc    |   Date

001   |  HB00001 | 23/4/12

002   |  HB00001 | 23/4/12

003   |  HB00001 | 23/4/12

004   |  HB00001 | 23/4/12

005   |  HB00001 | 23/4/12

From docx2:

Units |  Doc    |   Date

010   |  HB00002 | 2/6/16

011   |  HB00002 | 2/6/16

Final output:

Units |  Doc    |   Date

001   |  HB00001 | 23/4/12

002   |  HB00001 | 23/4/12

003   |  HB00001 | 23/4/12

004   |  HB00001 | 23/4/12

005   |  HB00001 | 23/4/12

010   |  HB00002 | 2/6/16

011   |  HB00002 | 2/6/16

Any help would be appreciated

Answer 1

The error is because the lengths of the columns are not the same. The moment the second file is processed, it will be trying to set the columns to values that have a different length to the first file. You cannot assign a column with values that are different to the existing columns.

Since you want the final df to have columns ['Units', 'Doc', 'Date'] , what you can do is to create a blank df and just append the new results to it. Use ignore_index=True to just append it below without trying to match row indexes.

import re
import docx2txt
import pandas as pd
import glob


final_df = pd.DataFrame()

for file in glob.glob("*.docx"):
    text = docx2txt.process(file)
    a1 = text.split()
    d2 = a1[37]
    doc2 = re.findall("HB0....", text)
    units2 = re.findall("00[0-9]...", text)

    # because columns are different length, create them as separate df and concat them
    df2 = pd.DataFrame()
    unit_df = pd.DataFrame(units2)
    doc_df = pd.DataFrame(doc2[0])
    date_df = pd.DataFrame(d2)
    # join them by columns. Any blanks will become NaN, but that's because your data has uneven lengths 
    df2 = pd.concat([df2, unit_df, doc_df, date_df], axis=1)

    # at the end of the loop, append it to the final_df
    final_df = pd.concat([final_df, df2], ignore_index=True)

print(final_df)

Answer 2

My suggestion is to first build a dict with the contents and create the DataFrame in the end:

import re
import docx2txt
import pandas as pd
import glob

columns = ['Units', 'Doc', 'Date']

data = {col: [] for col in columns}

for file in glob.glob("*.docx"):
    text = docx2txt.process(file)
    a1=text.split()
    d2=a1[37]
    doc2=re.findall("HB0....",text)
    units2=re.findall("00[0-9]...",text) 
    data['Units'].extend(units2)
    data['Doc'].extend(doc2[0])
    data['Date'].extend(d2)

df2 = pd.DataFrame(data)