Metpy Static Stability - MemoryError: Unable to allocate 13.4 GiB for an array with shape (19, 1825, 180, 288) and data type float64

Question

I have downloaded GFDL-ESM model global data for temperature for 2010-2014 period of around 3.5 GB. I want to calcuate static stability using the METPY library. My dataset looks like this Temperature Data

---

The code I am using is as follows:

import xarray as xr
import numpy as np
import pandas as pd
import geopandas as gp
import matplotlib.pyplot as plt
import cartopy
import cartopy.crs as ccrs
from cartopy import feature as cf
import netCDF4 as nc
import seaborn as sns
import glob
from datetime import datetime, timedelta
import cartopy.feature as cfeature
import matplotlib.gridspec as gridspec
import metpy.calc as mpcalc
from metpy.units import units
from metpy.plots.declarative import *
from netCDF4 import num2date
import scipy.ndimage as ndimage
from scipy.ndimage import gaussian_filter
from siphon.ncss import NCSS
from metpy.cbook import get_test_data
from metpy.interpolate import cross_section
import math

all=glob.glob(" {path} /ta_Eday_GFDL-ESM4_historical_r1i1p1f1_gr1_20100101-20141231.nc")
all

ds= xr.open_mfdataset(all).metpy.parse_cf()
ds

t=ds['ta']

pt = mpcalc.static_stability( t.plev*units.millibar , t*units.K )

---

After running this program the error I am getting is as follows:

--------
MemoryError                               Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_14188\3773388047.py in <cell line: 1>()
----> 1 pt = mpcalc.static_stability( t.plev*units.millibar , t*units.K )

~\miniconda3\lib\site-packages\metpy\xarray.py in wrapper(*args, **kwargs)
   1542                     )
   1543 
-> 1544         return func(*bound_args.args, **bound_args.kwargs)
   1545     return wrapper

~\miniconda3\lib\site-packages\metpy\xarray.py in wrapper(*args, **kwargs)
   1233 
   1234             # Evaluate inner calculation
-> 1235             result = func(*bound_args.args, **bound_args.kwargs)
   1236 
   1237             # Wrap output based on match and match_unit

~\miniconda3\lib\site-packages\metpy\units.py in wrapper(*args, **kwargs)
    294         def wrapper(*args, **kwargs):
    295             _check_units_inner_helper(func, sig, defaults, dims, *args, **kwargs)
--> 296             return func(*args, **kwargs)
    297 
    298         return wrapper

~\miniconda3\lib\site-packages\metpy\calc\thermo.py in static_stability(pressure, temperature, vertical_dim)
   3125 
   3126     """
-> 3127     theta = potential_temperature(pressure, temperature)
   3128 
   3129     return - mpconsts.Rd * temperature / pressure * first_derivative(

~\miniconda3\lib\site-packages\metpy\xarray.py in wrapper(*args, **kwargs)
   1233 
   1234             # Evaluate inner calculation
-> 1235             result = func(*bound_args.args, **bound_args.kwargs)
   1236 
   1237             # Wrap output based on match and match_unit

~\miniconda3\lib\site-packages\metpy\units.py in wrapper(*args, **kwargs)
    294         def wrapper(*args, **kwargs):
    295             _check_units_inner_helper(func, sig, defaults, dims, *args, **kwargs)
--> 296             return func(*args, **kwargs)
    297 
    298         return wrapper

~\miniconda3\lib\site-packages\metpy\calc\thermo.py in potential_temperature(pressure, temperature)
    137 
    138     """
--> 139     return temperature / exner_function(pressure)
    140 
    141 

~\miniconda3\lib\site-packages\metpy\xarray.py in wrapper(*args, **kwargs)
   1233 
   1234             # Evaluate inner calculation
-> 1235             result = func(*bound_args.args, **bound_args.kwargs)
   1236 
   1237             # Wrap output based on match and match_unit

~\miniconda3\lib\site-packages\metpy\units.py in wrapper(*args, **kwargs)
    294         def wrapper(*args, **kwargs):
    295             _check_units_inner_helper(func, sig, defaults, dims, *args, **kwargs)
--> 296             return func(*args, **kwargs)
    297 
    298         return wrapper

~\miniconda3\lib\site-packages\metpy\calc\thermo.py in exner_function(pressure, reference_pressure)
     95 
     96     """
---> 97     return (pressure / reference_pressure).to('dimensionless')**mpconsts.kappa
     98 
     99 

~\miniconda3\lib\site-packages\pint\quantity.py in __truediv__(self, other)
   1339 
   1340     def __truediv__(self, other):
-> 1341         return self._mul_div(other, operator.truediv)
   1342 
   1343     def __rtruediv__(self, other):

~\miniconda3\lib\site-packages\pint\quantity.py in wrapped(self, *args, **kwargs)
    137         elif isinstance(other, list) and other and isinstance(other[0], type(self)):
    138             return NotImplemented
--> 139         return f(self, *args, **kwargs)
    140 
    141     return wrapped

~\miniconda3\lib\site-packages\pint\quantity.py in wrapped(self, *args, **kwargs)
    117 def ireduce_dimensions(f):
    118     def wrapped(self, *args, **kwargs):
--> 119         result = f(self, *args, **kwargs)
    120         try:
    121             if result._REGISTRY.auto_reduce_dimensions:

~\miniconda3\lib\site-packages\pint\quantity.py in _mul_div(self, other, magnitude_op, units_op)
   1311             other = other.to_root_units()
   1312 
-> 1313         magnitude = magnitude_op(new_self._magnitude, other._magnitude)
   1314         units = units_op(new_self._units, other._units)
   1315 

MemoryError: Unable to allocate 13.4 GiB for an array with shape (19, 1825, 180, 288) and data type float64

Please help.

Answer 1

As indicated by the MemoryError , the problem is that the calculation you're requesting needs to allocate a large 13GB array, which isn't fitting in the memory on your system.

When you use open_mfdataset , xarray lazily loads the data, meaning it doesn't actually read all of the values into memory until they are requested.

Unfortunately, MetPy's calculations operate eagerly, meaning when you call static_stability on this full dataset, it's going to iterate over all of these values and needs to store the full final result into memory.

The Dask library tries to address this by allowing for "chunking" arrays that are bigger than can fit in memory and smartly handling computations. Unfortunately, while xarray can use Dask internally , MetPy's support for Dask right now is not particularly robust ; I'm not sure how well 1static_stability` would work when given a large Dask array.

If Dask doesn't work here with MetPy, one work-around is to only calculate on a subset of the data (eg whatever size you'd be doing analysis on, like a single time or a single vertical level) and iterate over these subsets.