Work with multiple netCDF files/variables in python

Question

I have around 4TB MERIS time series data which comes in netCDF format.

So I have a lot netCDF files containing several 'variables'. NetCDF format is new to me and although I've read a lot about netCDF processing I don't get an idea of how to do it. This question ' Combining a large amount of netCDF files ' deals somehow with my problem but I did not get there. My approach was to first mosaic, then stack and lately take the mean out of every pixel.

One file contains the following 32 variables

Here's additional the ncdump output of one .nc file of one day: http://www.filedropper.com/ncdumpoutput

I managed to read the files, extract the variables I want (variable # 32) and put them into a list using the following code

l = list()
for i in files_in:

# read netCDF file
dset = nc.Dataset(i, mode = 'r')

# save variables
var = dset.variables['vegetation_index_mean'][:]

# write all temp loop outputs in a list
l.append (var)

# close netCDF file
dset.close()

The list now contains 24 'masked_arrays' of different locations of the same date . Every time I want to print the contents of the list my Spyder freezes. Every command I run afterwards Spyder first freezes for five sec before starting.

My goal is to make a time series analysis for a specific time frame (every date stored in a single .nc file). So my plan was to mosaic (is this possible?) the variables in the list (treating them as raster bands), process additional dates and take the mean for every pixel (1800 x 1800 ).

Maybe my whole approach is wrong? Can I treat these 'variables' like raster bands?

Answer 1

I'm not sure if the following answer may respond to your needs, as this procedure is designed in order to process timeseries, is pretty manual and furthermore you have 4Tb of data...

Thus I apologize myself if this doesn't help.

This is for Python 2.7:

First import all the modules needed:

import tkFileDialog 
from netCDF4 import Dataset
import matplotlib.pyplot as plt

Second parse multiple nc files:

n = []
filename = {}
filename = tkFileDialog.askopenfilenames()  
filename = list(filename)
n = len(filename)

Third read nc files and classify data and metadata within dictionaries using a loop:

wtr_tem = {}  # create empty arrays for variable sea water temperature
fh = {}       # create empty arrays for filehandler and variables nc file
vars = {}

for i in range(n):
    filename[i]=filename[i].decode('unicode_escape').encode('ascii','ignore') # remove unicode in order to execute the following command
    filename1 = ''.join(filename[i]) # converts list to string
    fh[i] = Dataset(filename1, mode='r') #create the file handle
    vars[i] = fh[i].variables.keys()  #returns a list with the variables of the file

    wtr_tem[i] = fh[i].variables['WTR_TEM']

    #plot variables in different figures
    plt.plot(wtr_tem[i],'r-')

    plt.xlabel(fh[i].title) #add specific title from each nc file
    plt.show()

I hope it may help to somebody.