Pandas timeseries dataframe with multiple headers

Question

I'm trying to create a dataframe with timeseries data with multiple column headers, but I'm new to python and pandas. The data frame is one day of hourly data down and 1 million service points across. I can create the basic dataframe with the following:

intdata = pd.DataFrame(np.random.randint(0,1000,size=(24, 1000000)),
                   index=pd.date_range('2018-01-01',periods=24, freq='H'))

Index                 0   1   2   3   4   5   6   7 ... 1000000
2018-01-01 00:00:00 458 352 905 176 801 438 193 172         123
2018-01-01 01:00:00 68  313 465 460 960 487 574 335         123
2018-01-01 02:00:00 769 984 253 143 592 741 709 660         123
2018-01-01 03:00:00 316 684 195 660 602 200 228 748         123
2018-01-01 04:00:00 201 947 514 696 12  288 577 957         123
2018-01-01 05:00:00 235 118 746 880 909 365 233  57         123
...

I'd like the index row to be followed by a set of characteristics for each point like:

Service Point         0   1   2   3   4   5   6   7 **** my index
Characteristic_1      A   A   C   B   A   D   B   C **** characteristics needed
Characteristic_2      X   Y   Y   Z   Z   J   Q   J
2018-01-01 00:00:00 458 352 905 176 801 438 193 172
2018-01-01 01:00:00 68  313 465 460 960 487 574 335
...

How do I do that?

Thanks

Answer 1

pandas.MultiIndex.from_tuples

You need to specify what the levels will be.

tups = [(0, 'A', 'X'), (1, 'A', 'Y'), (2, 'C', 'Y'), (3, 'B', 'Z')]
mcol = pd.MultiIndex.from_tuples(
    tups, names=['Service Point', 'Characteristic_1', 'Characteristic_2'])

Then include these with the specification of the DataFrame

intdata = pd.DataFrame(
    np.random.randint(0,1000,size=(10, 4)),
    index=pd.date_range('2018-01-01',periods=10, freq='H'),
    columns=mcol
)

intdata

Service Point          0    1    2    3
Characteristic_1       A    A    C    B
Characteristic_2       X    Y    Y    Z
2018-01-01 00:00:00  400  800  426  433
2018-01-01 01:00:00  920  123  250  113
2018-01-01 02:00:00  319  300  187   33
2018-01-01 03:00:00  673  230  696  472
2018-01-01 04:00:00  703  766  962  796
2018-01-01 05:00:00  322  295  414  734
2018-01-01 06:00:00  987   38  400  848
2018-01-01 07:00:00  350  275  494  833
2018-01-01 08:00:00  677   58  335  293
2018-01-01 09:00:00  284  195  742  355

---

If you have the levels in existing lists, you can use zip

chr1 = [*'AACBADBC']
chr2 = [*'XYYZZJQJ']
tups = [*zip(range(8), chr1, chr2)]

mcol = pd.MultiIndex.from_tuples(
    tups, names=['Service Point', 'Characteristic_1', 'Characteristic_2'])
tidx = pd.date_range('2018-01-01',periods=10, freq='H')
data = np.random.randint(0, 1000, size=(len(tidx), len(mcol)))

intdata = pd.DataFrame(data, tidx, mcol)

intdata

Service Point          0    1    2    3    4    5    6    7
Characteristic_1       A    A    C    B    A    D    B    C
Characteristic_2       X    Y    Y    Z    Z    J    Q    J
2018-01-01 00:00:00  311  306  868   48  894  584  989  548
2018-01-01 01:00:00  848  170  592  640  638  400  112  642
2018-01-01 02:00:00  906  660  883  149  907  848  247  875
2018-01-01 03:00:00  461  432  479  733  979  540  311   86
2018-01-01 04:00:00  849  471  480  836  834  235  901   22
2018-01-01 05:00:00  758  193   45  405  739  818   81  577
2018-01-01 06:00:00  752  647  799  688  588  496   37  504
2018-01-01 07:00:00  380  785  750  975  960  535  971  257
2018-01-01 08:00:00  187  422  915  863  290  483  423  473
2018-01-01 09:00:00  270  144  749  710  983  755  839  709

---

pandas.MultiIndex.from_arrays

But then again, if you have the levels already in separate lists, you don't need to zip them yourself

chr1 = [*'AACBADBC']
chr2 = [*'XYYZZJQJ']

mcol = pd.MultiIndex.from_arrays(
    [range(8), chr1, chr2],
    names=['Service Point', 'Characteristic_1', 'Characteristic_2'])

tidx = pd.date_range('2018-01-01',periods=10, freq='H')

data = np.random.randint(0, 1000, size=(len(tidx), len(mcol)))

intdata = pd.DataFrame(data, tidx, mcol)

intdata

Service Point          0    1    2    3    4    5    6    7
Characteristic_1       A    A    C    B    A    D    B    C
Characteristic_2       X    Y    Y    Z    Z    J    Q    J
2018-01-01 00:00:00  311  306  868   48  894  584  989  548
2018-01-01 01:00:00  848  170  592  640  638  400  112  642
2018-01-01 02:00:00  906  660  883  149  907  848  247  875
2018-01-01 03:00:00  461  432  479  733  979  540  311   86
2018-01-01 04:00:00  849  471  480  836  834  235  901   22
2018-01-01 05:00:00  758  193   45  405  739  818   81  577
2018-01-01 06:00:00  752  647  799  688  588  496   37  504
2018-01-01 07:00:00  380  785  750  975  960  535  971  257
2018-01-01 08:00:00  187  422  915  863  290  483  423  473
2018-01-01 09:00:00  270  144  749  710  983  755  839  709