How to convert pandas dataframe to 3D Panel

Question

I am reading data from a CSV file which contains weather data for a network of buoys situated off the coast of Ireland. it is a time series dataset with hourly readings for each buoy. I want to create a 3D structure where there is a dataframe for each buoy, containing the columns of the weather conditions, indexed by the date and time.

I would like to be able to access the data via the following syntax:

df['column']['anotherColumn']

I'm aware that pandas has a deprecated Panel class, but I can't work out how to do this otherwise.

Any help would be appreciated, thanks!

Answer 1

The pandas Panel was deprecated in favour of DataFrame with multi-level index. To quote from the pandas documentation :

Hierarchical / Multi-level indexing is very exciting as it opens the door to some quite sophisticated data analysis and manipulation, especially for working with higher dimensional data. In essence, it enables you to store and manipulate data with an arbitrary number of dimensions in lower dimensional data structures like Series (1d) and DataFrame (2d).

Here's a quick example of a DataFrame with MultiIndex used to represent a three-dimensional data set:

In [1]: multi_index = pd.MultiIndex.from_arrays([
   ...:     ['buoy1', 'buoy1', 'buoy2', 'buoy2', 'buoy3', 'buoy3', 'buoy4', 'buoy4'],
   ...:     ['wind', 'water', 'wind', 'water', 'wind', 'water', 'wind', 'water'],
   ...: ])

In [2]: df = pd.DataFrame(np.random.randn(3, 8), columns=multi_index)

In [3]: df
Out[3]:
      buoy1               buoy2               buoy3               buoy4
       wind     water      wind     water      wind     water      wind     water
0  1.082442 -0.148975 -0.372837  0.075599  1.681150  0.910194  0.157064  0.183764
1 -0.019759  1.782505 -1.092751  0.324313 -2.217671  0.349224  1.085250 -0.715607
2 -1.308382 -0.994506 -0.306874  0.517858  1.356037 -0.024291  0.085105 -0.073061

Subsequently you can slice down to a 2D section of your data set like so:

In [4]: df['buoy3']
Out[4]:
       wind     water
0  1.681150  0.910194
1 -2.217671  0.349224
2  1.356037 -0.024291

And you can slice down to a 1D section (ie single column) of your data set like so:

In [5]: df['buoy3']['water']
Out[5]:
0    0.910194
1    0.349224
2   -0.024291
Name: water, dtype: float64