使用from_dict预先添加而不是在熊猫中附加NaN

Question

I have a pandas dataframe that I am reading from a defaultdict in Python, but some of the columns have different lengths. Here is what the data might look like:

Date      col1    col2    col3    col4    col5
01-01-15  5       12      1      -15      10
01-02-15  7       0       9       11      7
01-03-15          6       1       2       18
01-04-15          9       8       10
01-05-15         -4               7
01-06-15         -11             -1
01-07-15          6               

And I am able to pad the blanks with NaN s like so:

pd.DataFrame.from_dict(pred_dict, orient='index').T

Which gives:

Date      col1    col2    col3    col4    col5
01-01-15  5       12      1      -15      10
01-02-15  7       0       9       11      7
01-03-15  NaN     6       1       2       18
01-04-15  NaN     9       8       10      NaN
01-05-15  NaN    -4       NaN     7       NaN
01-06-15  NaN    -11      NaN    -1       NaN
01-07-15  NaN     6       NaN     NaN     NaN

However, what I'm really looking for is a way to prepend NaN s instead of appending them to the end, so that the data looks like this:

Date      col1    col2    col3    col4    col5
01-01-15  NaN     12      NaN     NaN     NaN
01-02-15  NaN     0       NaN    -15      NaN
01-03-15  NaN     6       NaN     11      NaN
01-04-15  NaN     9       1       2       NaN
01-05-15  NaN    -4       9       10      10
01-06-15  5      -11      1       7       7
01-07-15  7       6       8      -1       18

Is there an easy way to do this?

You can recreate the dictionary with this code:

import pandas as pd
from collections import defaultdict

d = defaultdict(list)
d["Date"].extend([
    "01-01-15", 
    "01-02-15", 
    "01-03-15", 
    "01-04-15", 
    "01-05-15",
    "01-06-15",
    "01-07-15"
])
d["col1"].extend([5, 7])
d["col2"].extend([12, 0, 6, 9, -4, -11, 6])
d["col3"].extend([1, 9, 1, 8])
d["col4"].extend([-15, 11, 2, 10, 7, -1])
d["col5"].extend([10, 7, 18])

Answer 1

You can use Series.shift to induce the Series/DataFrame. Unfortunately you can't pass in array of periods - you must shift each column by an integer value.

s = df.isnull().sum()
for col, periods in s.iteritems():
    df[col] = df[col].shift(periods)

Answer 2

A little modification on the itertools solution to your earlier question :

pd.DataFrame(list(itertools.zip_longest(*[reversed(i) for i in d.values()]))[::-1], columns=d.keys()).sort_index(axis=1)
Out[143]: 
       Date  col1  col2  col3  col4  col5
0  01-01-15   NaN    12   NaN   NaN   NaN
1  01-02-15   NaN     0   NaN -15.0   NaN
2  01-03-15   NaN     6   NaN  11.0   NaN
3  01-04-15   NaN     9   1.0   2.0   NaN
4  01-05-15   NaN    -4   9.0  10.0  10.0
5  01-06-15   5.0   -11   1.0   7.0   7.0
6  01-07-15   7.0     6   8.0  -1.0  18.0

Answer 3

Reverse every list in the dictionary:

for k, v in d.iteritems():
    d[k] = v[::-1]

df = pd.DataFrame.from_dict(d, orient='index').T.set_index('Date').sort_index(1).sort_index().astype(float)

Answer 4

Here's a vectorized approach that uses pd.DataFrame.from_dict to get the dataframe for the usual case. Once we have a regular 2D data available, it becomes easy enough to flip and mask and get to our desired output dataframe in a vectorized manner.

The implementation is listed below -

# Get the normal case output
df = pd.DataFrame.from_dict(d, orient='index').T

# Use masking to flip and select flipped elements to re-create expected df
colmask = df.columns!='Date'
arr = np.array(df.ix[:,colmask].values, dtype=np.float).T
mask = ~np.isnan(arr)
out_arr = np.full(mask.shape,np.nan)
out_arr[mask[:,::-1]] = arr[mask]
df.ix[:,colmask] = out_arr.T

Sample run -

In [209]: d.values()
Out[209]: 
[[-15, 11, 2, 10, 7, -1],
 [10, 7, 18],
 [12, 0, 6, 9, -4, -11, 6],
 [1, 9, 1, 8],
 [5, 7],
 ['01-01-15',
  '01-02-15',
  '01-03-15',
  '01-04-15',
  '01-05-15',
  '01-06-15',
  '01-07-15']]

In [210]: df
Out[210]: 
  col4 col5 col2 col3 col1      Date
0  NaN  NaN   12  NaN  NaN  01-01-15
1  -15  NaN    0  NaN  NaN  01-02-15
2   11  NaN    6  NaN  NaN  01-03-15
3    2  NaN    9    1  NaN  01-04-15
4   10   10   -4    9  NaN  01-05-15
5    7    7  -11    1    5  01-06-15
6   -1   18    6    8    7  01-07-15