如何在数据立方体中搜索具有除 NaN 以外的值的最近邻点？

Question

I am working with a datacube such as data[x,y,z]. Each point is a velocity through time, and the [x,y] grid corresponds to coordinates. If I pick a point of coordinates x and y, it is likely that the timeseries is incomplete (with some NaNs). I created a function which searches for the closest neighbor with a value, and replaces the NaN of my xy point with it. However I want to know if there is a more efficient way to code something which does the same?

Joined to this message is a photo of how the function evaluates the neighbors. The number of each point represents its rank (5 is the 5th neighbor evaluated).

I tried something like this:

Let's say that I have a datacube of 10x10x100 (100 is the timeseries):

import math
import numpy as np

Vel = np.random.rand(10,10,100)
Vel[4:7,4:7,0:10] = np.nan

x = 5
y = 5

Vpoint = Vel[5,5,:]


for i in range(0,len(Vpoint)):
        
        xx = x
        yy = y
        
        
        if math.isnan(Vel[xx,yy,i]) == True:
        
            for n in range(0,50):
                
                n = n + 1
            
                if n > 10:
                    raise Exception("The interpolation is way too far") 
            
                xx = x + n
                yy = y
            
                if math.isnan(Vel[xx,yy,i]) == False:
                    Vpoint[i] = Vel[xx,yy,i]
                    break
            
                xx = x-n
                if math.isnan(Vel[xx,yy,i]) == False:
                    Vpoint[i] = Vel[xx,yy,i]
                    break
            
                xx = x
                yy = y + n
                if math.isnan(Vel[xx,yy,i]) == False:
                    Vpoint[i] = Vel[xx,yy,i]
                    break
                
                yy = y-n
                if math.isnan(Vel[xx,yy,i]) == False:
                    Vpoint[i] = Vel[xx,yy,i]
                    break
            
                for p in range(1,n):
                    
                    xx = x+p
                    if math.isnan(Vel[xx,yy,i]) == False:
                        Vpoint[i] = Vel[xx,yy,i]
                        break
                    xx = x-p
                    if math.isnan(Vel[xx,yy,i]) == False:
                        Vpoint[i] = Vel[xx,yy,i]
                        break
                    
                    
                    for p in range(1,n):
                        
                        
                        yy = y+n
                        xx = x+p
                        
                        if math.isnan(Vel[xx,yy,i]) == False:
                            Vpoint[i] = Vel[xx,yy,i]
                            break
                        xx = x-p
                        if math.isnan(Vel[xx,yy,i]) == False:
                            Vpoint[i] = Vel[xx,yy,i]
                            break    
                        
                        
                        yy = y-n
                        xx = x+p
                        
                        if math.isnan(Vel[xx,yy,i]) == False:
                            Vpoint[i] = Vel[xx,yy,i]
                            break
                        xx = x-p
                        if math.isnan(Vel[xx,yy,i]) == False:
                            Vpoint[i] = Vel[xx,yy,i]
                            break
                        
                        xx = x+n
                        yy = y+p
                        
                        if math.isnan(Vel[xx,yy,i]) == False:
                            Vpoint[i] = Vel[xx,yy,i]
                            break
                        yy = y-p
                        if math.isnan(Vel[xx,yy,i]) == False:
                            Vpoint[i] = Vel[xx,yy,i]
                            break    
                        
                        
                        xx = x-n
                        yy = y-p
                        
                        if math.isnan(Vel[xx,yy,i]) == False:
                            Vpoint[i] = Vel[xx,yy,i]
                            break
                        yy = y-p
                        if math.isnan(Vel[xx,yy,i]) == False:
                            Vpoint[i] = Vel[xx,yy,i]
                            break

        print(n,xx,yy)

Ps: in reality my timeseries is close to 330x300x38000, and the closest non-nan neighbor should change every time.

Answer 1

Here is what I came up with:

import numpy as np
from scipy import interpolate

Vel = np.random.rand(10,10,100) 
Vel[4:7,4:7,0:10] = np.nan
Vel[4:7,4:7,20:30] = np.nan

def gap_filling(vect, interpolation):

    time = np.arange(0, np.shape(vect)[0])
    mask = np.isfinite(vect)
    f = interpolate.interp1d(time[mask], vect[mask], 
                             kind=interpolation, bounds_error=False)

    vect_filled = np.copy(vect)
    vect_filled[np.isnan(vect)] = f(time[np.isnan(vect)])

    return vect_filled

Vel_filled_nn = np.apply_along_axis(gap_filling, -1, Vel, 'nearest')

Vel_filled_li = np.apply_along_axis(gap_filling, -1, Vel, 'linear')

I create an interpolation function based on the available data through time, then map it onto the missing values and that for each time series of the data set.

But because I know for which application you are developing this code (data analysis in Earth sciences), I'd advise you to use an interpolation instead of a nearest neighbour (here Vel_filled_li ). The results on one of the time series:

import matplotlib.pyplot as plt

plt.figure()
plt.plot(Vel_filled_nn[6, 6, :], 'o-', label='nearest neighbour')
plt.plot(Vel_filled_li[6, 6, :], 'o-', label='linear interpolation')
plt.plot(Vel[6, 6, :], 'o-', label='raw')
plt.legend(loc='upper right')
plt.xlabel('Time', fontsize=15)
plt.ylabel('Variable', fontsize=15)

It is only a base and can/should be vectorised, using the axis parameter of interpolate.interp1d .