如何提高阅读能力，用python转换300个xml文件

Question

I have 300 xml files on my folder.

Each one gives me ~ 1.657 rows when I transform to a dataframe.

The code bellow is taking too much time.

Using R I did in ~200seconds.

The function: function_from_xml_pddataframe(xmlfile) generates the df_xml_ dataframe

What am I doing wrong?

How can I improve this process?

   import os
    all_dfs = pd.DataFrame()

    
    
    for file in tqdm("/data"):
    
        if file.endswith(".xml"):
          function_from_xml_pddataframe(xmlfile)
          
          df_created = df_xml

          list_of_dataframes.append(df_created)



all_dfs = pd.concat(list_of_dataframes)

Answer 1

Without a reproducible example it's hard to be sure. Even replacing your file loop and function_from_xml_pddataframe your code won't run as it is:

• list_of_dataframes needs to be defined before you can append to it.
• If you meant line 2 to read: list_of_dataframes = pd.DataFrame() your code would run slowly and then error at the last line all_dfs = pd.concat(list_of_dataframes) .

Assuming you wanted something like this (which is slow, but runs):

import pandas as pd
from scipy.stats import uniform

list_of_dfs = pd.DataFrame()

for i in range(10):
    list_of_dfs.append(pd.DataFrame({'a': uniform.rvs(0, 1, 1000), 'b': uniform.rvs(0, 1, 1000)}))

# this line will cause an error
pd.concat(list_of_dfs)

It looks like you're setting up all_dfs (or list_of_dataframes as it should be) as a pandas dataframe. I think what you're trying to do here is set it up as a list. Try changing line 2 to list_of_dataframes = [] :

import pandas as pd
from scipy.stats import uniform

list_of_dataframes = []
for i in range(10):
    list_of_dataframes.append(pd.DataFrame({'a': uniform.rvs(0, 1, 1000), 'b': uniform.rvs(0, 1, 1000)}))

pd.concat(list_of_dataframes)

The second version is ~10x faster.

TL;DR
Make sure you store your separate pandas dataframes in a list - appending to a dataframe is slow.