How to solve pandas.get_dummies Exception: Data must be 1-dimensional

Question

I am trying to read the wages dataset Wages.csv . Then tried to bin the columns. But I am getting an exception that shows data must be 1- dimensional

The codes has been reproduced below and dataset link given.

# import modules
  import pandas as pd
  import numpy as np
  import statsmodels.api as sm
  import matplotlib.pyplot as plt 
  %matplotlib inline

  # read data_set
  data = pd.read_csv("Wage.csv")
  data.head()


  data_x = data['age']
  data_y = data['wage']

  # Dividing data into train and validation datasets
  from sklearn.model_selection import train_test_split
  train_x, valid_x, train_y, valid_y = train_test_split(data_x, data_y, test_size=0.33, random_state = 1)

  # Dividing the data into 4 bins
    df_cut, bins = pd.cut(train_x, 4, retbins=True, right=True)
    df_cut.value_counts(sort=False)

    df_steps = pd.concat([train_x, df_cut, train_y], keys=['age','age_cuts','wage'], axis=1)

    # Create dummy variables for the age groups
    df_steps_dummies = pd.get_dummies(df_cut)
    df_steps_dummies.head()



   df_steps_dummies.columns = ['17.938-33.5','33.5-49','49-64.5','64.5-80'] 

   # Fitting Generalised linear models
    fit3 = sm.GLM(df_steps.wage, df_steps_dummies).fit()

    # Binning validation set into same 4 bins
    bin_mapping = np.digitize(valid_x, bins) 
    X_valid = pd.get_dummies(bin_mapping)

I am getting an exception Exception: Data must be 1-dimensional

Answer 1

If you look at the data it is of the form: [1] [2] … [3]

You need to get it to something like [1 2 … 3]

Flattening the data into a single list then dropping it back into an np.array works.

For example:

Code:

def binMapping(x): 
  flat = [] 
  prestep = np.digitize(x, bins)
  for sublist in prestep:
    for ele in sublist:
    flat.append(ele)  
  return np.array(flat)

bin_mapping = binMapping(valid_x)
X_valid = pd.get_dummies(bin_mapping)

This works. I'm sure there's a better way to do it.