Error in list comprehension with `locals()` in python

Question

I have run a python script that created multiple variables. Now I want to iterate over a few dataframes (created by the script) matching a specific pattern and perform simple operations on them. Initially I want to get the number of rows (with shape() ) of each of the dataframes in list_dfs , as shown below:

['FAILEDRuns_0112',
 'FAILEDRuns_0121',
 'FAILEDRuns_0126',
 'FAILEDRuns_0129',
 'FAILEDRuns_0131',
 'FAILEDRuns_0134',
 'FAILEDRuns_0135',
 'FAILEDRuns_0137',
 'FAILEDRuns_0142',
 'FAILEDRuns_0153',
 'FAILEDRuns_0165',
 'FAILEDRuns_0171',
 'FAILEDRuns_0175']

In fact if I do:

for i in list(filter(failed_runs_finder.findall, dir())):
    print(locals()[i].shape[0])

I get the number of rows printed onto the screen:

1
0
0
0
1
0
0
0
0
0
0
0
0

Which contains the information that I need, though not in the format that I want. Eventually what I need to know is the number of 1's and the number of zero's, so I thought about getting a list comprehension, to eventually compare the total sum (ie the number of 1's) with the length of the list ie the total number of elements.

However, if I do:

[locals()[i].shape[0] for i in list_dfs]

I get the following error:

KeyError: 'FAILEDRuns_0112'

I don't quite understand where the error is coming from. As far as I see, it is not in terms of syntax of list comprehensions.

Does it have anything to do with using locals() within a list comprehension?

My second option would be to build a df iteratively and get the sum, though I think it is simpler with list comprehension and I don't quite get where the error is coming from.

Answer 1

Try this instead if you really must rely on locals() :

[v.shape[0] for k, v in locals().items() if k in list_dfs]

However it would probably be a better approach, as suggested, to use a single dict to store all the names and DataFrame objects instead.

If you want to get the counts of rows:

from collections import Counter

cnt = Counter(v.shape[0] for k, v in locals().items() if k in list_dfs)

cnt[1]
# 2

cnt[0]    
# 11