数据框字典中的Pandas面板返回NaN

Question

I have a group of DataFrames that I'm trying to turn into a Panel. Here is my code:

# OPEN THE FILES INTO DATAFRAMES
filenames = ['Yahoo_2016-01-17.csv', 'Yahoo_2016-01-18.csv',
    'Yahoo_2016-01-19.csv','Yahoo_2016-01-23.csv','Yahoo_2016-01-27.csv',     
    'Yahoo_2016-02-05.csv', 'Yahoo_2016-02-06.csv', 'Yahoo_2016-02-09.csv',     
    'Yahoo_2016-02-11.csv', 'Yahoo_2016-02-13.csv', 'Yahoo_2016-02-15.csv', 
    'Yahoo_2016-02-16.csv', 'Yahoo_2016-02-29.csv']

dates = np.array(['2016-01-17', '2016-01-18', '2016-01-19', '2016-01-23', 
    '2016-01-27', '2016-02-05', '2016-02-06','2016-02-09', 
    '2016-02-11', '2016-02-13', '2016-02-15', '2016-02-16',
    '2016-02-29']).astype('datetime64[D]')

filepath = '/Users/RickS/Documents/Investing/Stock_files/GENERAL/'

dfs = [pd.read_csv(filepath+f) for f in filenames]

# Panel not working...
panel = pd.Panel(dict([(date, df) for date in dates for df in dfs]))
panel.swapaxes('major','minor')

However when I try to read the panel, all the values in each dataframe have turned into NaNs:

When I look at the dataframes individually they all look fine. Here is one of the csv files that gets imported into df: example_csv_file

One thing to note that may (or may not) be important is that the dtypes for each dataframe are not all the same:

In [24]: dfs[1].dtypes
Out[24]: 
Name                          object
Symbol                        object
Previous_Close               float64
Average_Daily_Volume           int64
Change_&_Percent_Change       object
Earnings/Share               float64
EPS_Estimate_Current_Year    float64
EPS_Estimate_Next_Quarter    float64
EPS_Estimate_Next_Year       float64
52-week_Low                  float64
52-week_High                 float64
EBITDA                        object
200-day_Moving_Average       float64
P/E_Ratio                    float64
PEG_Ratio                    float64
Short_Ratio                  float64
1_yr_Target_Price            float64
52-week_Range                 object
Date                          object
dtype: object

What am I doing wrong?

Answer 1

Reason for empty panel with all NaNs is your dates numpy array currently stored as datetime64 types. Apparently, the pandas panel object does not work well with underlying dictionary keys.

Simply remove the astype or even use a list or tuple which will render dates as string keys. But since dictionary keys are measured through days, each will be unique for your panel needs.

dates = np.array(['2016-01-17', '2016-01-18', '2016-01-19', '2016-01-23', 
                  '2016-01-27', '2016-02-05', '2016-02-06','2016-02-09', 
                  '2016-02-11', '2016-02-13', '2016-02-15', '2016-02-16',
                  '2016-02-29'])

dates = ['2016-01-17', '2016-01-18', '2016-01-19', '2016-01-23', 
         '2016-01-27', '2016-02-05', '2016-02-06','2016-02-09', 
         '2016-02-11', '2016-02-13', '2016-02-15', '2016-02-16',
         '2016-02-29']

However, this brings my earlier find. Currently, the list comprehension within the dict() function will return a panel of only the last data frame, repeated 13 times. Reason being the list comprehension below returns a total combination set between the dfs list and dates array with a length equal to the product of both collections: 13 X 13 (ie, cross join/cartesian join). Output below to see:

[(date, df) for date in dates for df in dfs]

Once you apply dict() to above, you force the 13 unique dates to carry the value of last df , essentially pulling in the last combination pairing .

Consider using zip() to iterate over each item of both collections together:

dfDict = {}
for f,d in zip(filenames, dates):    
    dfDict[d] = pd.read_csv(filepath+f)    

panel = pd.Panel(dfDict)

Or the shorter:

dfs = [pd.read_csv(filepath+f) for f in filenames] 
panel = pd.Panel(dict([i for i in zip(dates, dfs)]))