如何在一組切片中對Pandas系列的連續NaN值進行分組？

Question

我想將連續的NaN值合並到切片中。 是否有一種簡單的方法可以用numpy或pandas做到這一點？

l = [
    (996, np.nan), (997, np.nan), (998, np.nan),
    (999, -47.3), (1000, -72.5), (1100, -97.7),
    (1200, np.nan), (1201, np.nan), (1205, -97.8),
    (1300, np.nan), (1302, np.nan), (1305, -97.9),
    (1400, np.nan), (1405, -97.10), (1408, np.nan)
]
l = pd.Series(dict(l))

預期結果：

[
    (slice(996, 999, None), array([nan, nan, nan])),
    (999, -47.3),
    (1000, -72.5),
    (1100, -97.7),
    (slice(1200, 1202, None), array([nan, nan])),
    (1205, -97.8),
    (slice(1300, 1301, None), array([nan])),
    (slice(1302, 1303, None), array([nan])),
    (1305, -97.9),
    (slice(1400, 1401, None), array([nan])),
    (1405, -97.1),
    (slice(1408, 1409, None), array([nan]))
]

具有兩個維度的numpy數組也可以，而不是元組列表

更新2019/05/31 ：我剛剛意識到，如果我只使用字典而不是Pandas系列，那么algorythm會更有效率

Answer 1

你想要的是完整或角落情況，nan相等，每對的第一個元素是一個切片或一個單獨的值，第二個是np.array或單個值。

對於如此復雜的需求，我只依賴於普通的Python非向量化方式：

def trans(ser):
    def build(last, cur, val):
        if cur == last + 1:
            if np.isnan(val):
                return (slice(last, cur), np.array([np.nan]))
            else:
                return (last, val)
        else:
            return (slice(last, cur), np.array([val] * (cur - last)))
    last = ser.iloc[0]
    old = last_index = ser.index[0]
    resul = []
    for i in ser.index[1:]:
        val = ser[i]
        if ((val != last) and not(np.isnan(val) and np.isnan(last))) \
           or i != old + 1:
            resul.append(build(last_index, old + 1, last))
            last_index = i
            last = val
        old = i
    resul.append(build(last_index, old+1, last))
    return resul

它提供了接近預期結果的東西：

[(slice(996, 999, None), array([nan, nan, nan])),
 (999, -47.3),
 (1000, -72.5),
 (1100, -97.7),
 (slice(1200, 1202, None), array([nan, nan])),
 (1205, -97.8),
 (slice(1300, 1301, None), array([nan])),
 (slice(1302, 1303, None), array([nan])),
 (1305, -97.9),
 (slice(1400, 1401, None), array([nan])),
 (1405, -97.1),
 (slice(1408, 1409, None), array([nan]))]

Answer 2

cumsum的notnull是一個好主意，但是我們需要過濾掉每個子系列中的第一個非空值，這樣我們就可以對該對進行(cumsum, notnull) ：

# convert series to frame, 
# don't know why series only doesn't work
df = l.to_frame(name='val')

df['notnull'] = df['val'].notnull()
g = df.groupby([ df['notnull'].cumsum(), 'notnull']).val

[(v.index, v.values) for i, v in g]

日期：

[(Int64Index([996, 997, 998], dtype='int64'), array([nan, nan, nan])),
 (Int64Index([1200, 1201], dtype='int64'), array([nan, nan])),
 (Int64Index([1300, 1302, 1400, 1402], dtype='int64'),
  array([nan, nan, nan, nan])),
 (Int64Index([999], dtype='int64'), array([-47.3])),
 (Int64Index([1000], dtype='int64'), array([-72.5])),
 (Int64Index([1100], dtype='int64'), array([-97.7])),
 (Int64Index([1202], dtype='int64'), array([-97.1]))]

---

編輯：考慮連續索引並更新切片：

# convert group to slices
def get_slice(x):
    idx_min, idx_max = x.index.min(), x.index.max()

    if len(x) >1:
        return (slice(idx_min, idx_max+1), x.values)
    elif x.isna().any():
        return (slice(idx_min, idx_min+1), x.values)
    else:
        return (idx_min, x[idx_min])

df['notnull'] = df['val'].notnull()

# non-continuous indices
df['sep'] = (df.index != df.index.to_series().shift() + 1).cumsum()

g = df.groupby(['sep', df['notnull'].cumsum(), 'notnull']).val

g.apply(get_slice).values.tolist()

得到：

[(slice(996, 999, None), array([nan, nan, nan])),
 (999, -47.3),
 (1000, -72.5),
 (1100, -97.7),
 (slice(1200, 1202, None), array([nan, nan])),
 (1205, -97.8),
 (slice(1300, 1301, None), array([nan])),
 (slice(1302, 1303, None), array([nan])),
 (1305, -97.9),
 (slice(1400, 1401, None), array([nan])),
 (1405, -97.1),
 (slice(1408, 1409, None), array([nan]))]