How can I optimize this X-Ray selection cycle?

Question

I have the following code

facts = []
with tqdm(total=6022484) as pbar:
for lat in dp.lat:
    for lon in dp.lon:
        for time in dp.time:
            fact = {\
                    'datetime':datetime.datetime.fromtimestamp(float(time.values)/1000000000.),\
                    'loc':[float(lon.values),float(lat.values)],\
                    'temp':celsius(dp.sel(lat=lat.values,lon=lon.values,time=time.values).t2m.values),\
                    'rh':round(dp.sel(lat=lat.values,lon=lon.values,time=time.values).rh.values,1),\
                    'rain':round(dp.sel(lat=lat.values,lon=lon.values,time=time.values).rain.values,1)
                   }
            facts.append(fact)
            pbar.update()

making aprox. 100 iteration per second. Is it possible to do best?

Answer 1

Generally this approach is going to be extremely slow. Instead of iterating through these in python, you should use the standard functions which operate across the values in a vectorized way.

For example, dp.lat = dp.lat.astype('float') , or dp.rain = np.round(dp.rain, 0) .

This is a similar discussion: What is the most efficient way to loop through dataframes with pandas?