Logging progress of `enumerate` function in Python

Question

I am working on optimizing a Python (version 2.7; constrained due to old modules I need to use) script that is running on AWS and trying to identify exactly how many resources I need in the environment I am building. In order to do so, I am logging quite a bit of information to the console when the script runs and benchmarking different resource configurations.

One of the bottlenecks is a list with 32,000 items that runs through the Python enumerate function. This takes quite a while to run and my script is blind to its progress until it finishes. I assume enumerate is looping through the items in some fashion so thought I could inject logging or printing within that loop. I know this is 'hacky' but it will work fine for me for now since it is temporary while I run tests.

I can't find where the function runs from (I did find an Enumerate class in the numba module. I tried printing in there and it did not work). I know it is part of the __builtin__ module and am having trouble tracking that down as well and tried several techniques to find its exact location such as printing __builtin__.__file__ but to no avail.

My question is, a) can you all help me identify where the function lives and if this is a good approach? or b) if there is a better approach for this.

Thanks for your help!

Answer 1

I suggest you to use tqdm module. tqdm

tqdm module is used for visualization of the progress of an enumeration. It works with iterables (length known or unknown).

By extending tqdm class you can log required information after any required number of iterations. You can find more information in the tqdm class docstring.