熊猫和Numpy Fancy索引

Question

I am having a hard time with the following. I have a pandas N x D dataframe called p with some missing ( NAN ) values. I have another corresponding array indexed by D x K x T . I want to make a map of every panda's entry n,d in the data frame to a[d][k][p[n,d]] for all possible k resulting in N x D x K matrix. Can I have some help as to how to do this most efficiently with the Pandas and Numpy library?

I actually then take the N x D part of the final matrix and take the product along the columns leaving an N x K matrix. The final output can be (slowly) reproduced by the following:

    def generate_entry(i, j):
        result = np.prod([alpha[s][j][int(p.loc[i][s])] for s in range(num_features) if not isNaN(p.loc[i][s]) ])
        return result

    vgenerate_entry = np.vectorize(generate_entry)
    result = np.fromfunction(vgenerate_entry, shape=(len(p), k), dtype=int)

I think some use of pandas.get_dummies would be helpful for matrix multiplication but I can't quite figure that out.

The following is much faster:

    r = None
    for i in range(num_features):
        rel_data = pd.get_dummies(data.ix[:,i])
        rel_probs = alpha[i].T
        prod = rel_data.dot(rel_probs)
        prod[prod == 0] = 1
        if r is None:
            r = prod
        else:
            r = r.multiply(prod)

    r = r.as_matrix()
    r = r * pi
    posteriers = r / np.sum(r, axis=1)[:, np.newaxis]

Answer 1

Here's one approach to index into the NumPy array a with the pandas dataframe p that has NaNs , which are to be avoided and we are filling some value fillval in those places -

def fancy_indexing_avoid_NaNs(p, a, fillval = 1):
    # Extract values from p and get NaN mask
    pv = p.values
    mask = np.isnan(pv)

    # Get int version, replacing NaNs with some number, say 0
    p_idx = np.where(mask, 0, pv).astype(int)

    # FANCY-INDEX into array 'a' with those indices fron p
    a_indexed_vals = a[np.arange(D), np.arange(K)[:,None,None],p_idx]

    # FANCY-INDEX once more to replace the values set by NaNs as 1s, so
    # that in the prod-reduction later on they would have no effect
    a_indexed_vals[np.arange(K)[:,None,None],mask] = fillval
    return a_indexed_vals

That fillval would be application dependent. In this case, we are using prod , so a fillval=1 makes sense, which won't affect the results.

Original approach posted by OP -

def generate_entry(i, j):
    result = np.prod([a[s][j][int(p.loc[i][s])] for s in range(D) \
                                   if not np.isnan(p.loc[i][s]) ])
    return result

vgenerate_entry = np.vectorize(generate_entry)

Sample run -

In [154]: N,D,K,T = 3,4,5,6
     ...: a = np.random.randint(0,5,(D,K,T))
     ...: 
     ...: p = pd.DataFrame(np.random.randint(0,T,(N,D)).astype(float))
     ...: p.iloc[2,3] = np.nan
     ...: p.iloc[1,2] = np.nan
     ...: 

In [155]: result = np.fromfunction(vgenerate_entry, shape=(len(p), K), dtype=int)

In [156]: a_indexed_vals = fancy_indexing_avoid_NaNs(p, a)

In [157]: out = a_indexed_vals.prod(2).T

In [158]: np.allclose(out, result)
Out[158]: True