How do I store mutidimensional arrays contained in a dictionary in a python xarray?

Question

I have a dictionary containing numpy arrarys of varying sizes. All arrays have a common axis length (time) that I would like to store the data along.

For example:

arr1 = np.random.rand(239, 1)
arr2 = np.random.rand(239, 6)
arr3 = np.random.rand(239, 3, 7)
time = np.random.rand(239, 1)
d = {'A': arr1, 'B': arr2, 'C': arr3, 'time': time}

I need to be able to index and manipulate the data easily so my first inclination was to use a pandas.Panel to store the data, however, with the inconsistency in dimensions I have been unsuccessful.

Is a xarray.Dataset the right approach to take here to store my data, if so, how would that be best implemented?

Answer 1

Here's a pretty simple approach using standard pandas methods. For brevity and convenience I'm making your data smaller and putting into dataframes, but the concept is the same.

dr=pd.date_range('1-1-2017', periods=4, freq='d')
df1=pd.DataFrame( np.random.randn(4),   columns=['x'],     index=dr)
df2=pd.DataFrame( np.random.randn(4,2), columns=['y','z'], index=dr)

So df1 & df2 look like this:

                   x
2017-01-01 -0.705449
2017-01-02 -0.597631
2017-01-03 -0.844197
2017-01-04 -1.063895
                   y         z
2017-01-01 -0.288822 -0.343934
2017-01-02  1.072678  1.776767
2017-01-03 -0.606593  0.192280
2017-01-04  0.019401  2.007770

Re-configure like this:

df = df1.stack().append(df2.stack()).sort_index()

---

2017-01-01  x   -0.705449
            y   -0.288822
            z   -0.343934
2017-01-02  x   -0.597631
            y    1.072678
            z    1.776767
2017-01-03  x   -0.844197
            y   -0.606593
            z    0.192280
2017-01-04  x   -1.063895
            y    0.019401
            z    2.007770

And you can even convert from here to xarray with:

df.to_xarray()

Some quick notes:

• Panel is deprecated in favor of either xarrays or multi-indexes. I took the multi-index approach above but xarrays is another good option
• For a great theory of data organization, see Hadley Wickam's explanation of "tidy" data, which you can find here .