如何改善python脚本中的执行时间？

Question

I'm trying to remove values from files containing 3 dimensional arrays corresponding with the dimensions corresponding to [time][long][lat] .

I have separate files for each years worth of data. I have a list of observed data points T_obs for a duration of time start_time_index to end_time_index that I want to compare to the mean value over that time period in the file data.

The data sets contained in the files are sufficiently large that my code is running very slowly and I want to optimize my execution time. The code I have currently is below. Are there any ways I could significantly save time?

T_obs = [1.5, 3.6, 4.5]
start_time_index = [20, 300, 10]
end_time_index = [40, 328, 200]
long_obs = [45, 54, 180]
lat_obs = [34, 65, 32]
LE = np.zeros(len(T_obs))
t = 1984

for filename in os.listdir("C:\\Directory"):
    if filename.endswith(".nc"):
        print(filename)
        fh = Dataset("C:\\Directory %s"
                     % filename, 'r').variables['matrix']
        for i in range(0, len(long_obs)):
            if year[i] == t and start_time_index[i] > 0:
                LE_t = []
                for x in range(int(start_time_index[i]), int(end_time_index[i])):
                    LE_t = np.append(LE_t,float(fh[x][long_obs[i]+180][lat_obs[i]*-1+90])/10)
                LE[i] = np.mean(LE_t)
        t += 1
        continue
    else:
        continue 

Answer 1

Are you going to do this kind of work many times (with different values), or just once?

If the former, you can try to put your files data in a database (such as MySQL , which is simple to setup) and create indexes on the start and end time. This would make your reads mighty faster, as you would not need to do a full table scan (which is pretty much what you are doing by reading the whole file).

If you are doing it just once, then I just suggest you wait it out. There is no trivial way to make I/O (which is your bottleneck) less expensive and it seems like to make your checks / compare the data, you actually need to go through the whole dataset.