尝试重新索引列时 pyspark.pandas 出错

Question

I have a dataframe that is missing certain values and I want to generate records for those and input the value with 0.

My df looks like this:

import pyspark.pandas as ps
import databricks.koalas as ks
import pandas as pd

data = {'Region': ['Africa','Africa','Africa','Africa','Africa','Africa','Africa','Asia','Asia','Asia'],
         'Country': ['South Africa','South Africa','South Africa','South Africa','South Africa','South Africa','South Africa','Japan','Japan','Japan'],
         'Product': ['ABC','ABC','ABC','XYZ','XYZ','XYZ','XYZ','DEF','DEF','DEF'],
         'Year': [2016, 2018, 2019,2016, 2017, 2018, 2019,2016, 2017, 2019],
         'Price': [500, 0,450,750,0,0,890,19,120,3],
         'Quantity': [1200,0,330,500,190,70,120,300,50,80],
         'Value': [600000,0,148500,350000,0,29100,106800,74300,5500,20750]}

df = ps.DataFrame(data)

Some entries in this df are missing, such as the year 2017 for South Africa and the year 2018 for Japan .

I want to generate those entries and add 0 in the columns Quantity , Price and Value .

I managed to do this on a smaller dataset using pandas , however, when I try to implement this using pyspark.pandas , I get an error.

This is the code I have so far:

(df.set_index(['Region', 'Country','Product','Year'])
   .reindex(ps.MultiIndex.from_product([df['Region'].unique(), 
                                        df['Country'].unique(),
                                        df['Product'].unique(),
                                        df['Year'].unique()], 
                                       names=['Region', 'Country','Product','Year']), 
            fill_value=0)
   .reset_index())

Whenever I run it, I get the following issue:

PandasNotImplementedError: The method `pd.Series.__iter__()` is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead.

Any ideas why this might happen and how to fix it?

Answer 1

Ok so looking closer at the synatx for ps.MultiIndex, the variables have to be passed as list, so I had to add .tolist() after each column's unique values. Code below:

(df.set_index(['Region', 'Country','Product','Year'])
   .reindex(ps.MultiIndex.from_product([df['Region'].unique().tolist(), 
                                        df['Country'].unique().tolist(),
                                        df['Product'].unique().tolist(),
                                        df['Year'].unique().tolist()], 
                                       names=['Region', 'Country','Product','Year']), 
            fill_value=0)
   .reset_index())