从使用 pyspark 获得的数据帧中缩放数据

Question

I'm trying to scale some data from a csv file. I'm doing this with pyspark to obtain the dataframe and sklearn for the scale part. Here is the code:

from sklearn import preprocessing
import numpy as np
import pyspark

from pysparl.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.read.option('header','true').csv('flights,csv')
X_scaled = preprocessing.scale(df)

If I make the dataframe with pandas the scale part doesn't have any problems, but with spark I get this error:

ValueError: setting an array element with a sequence.

So I'm guessing that the element types are different between pandas and pyspark, but how can I work with pyspark to do the scale?

Answer 1

sklearn works with pandas dataframe. So you have to convert spark dataframe to pandas dataframe.

X_scaled = preprocessing.scale(df.toPandas())

Answer 2

You can use the "StandardScaler" method from "pyspark.ml.feature". Attaching a sample script to perform the exact pre-processing as sklearn,

Step 1:

from pyspark.ml.feature import StandardScaler

scaler = StandardScaler(inputCol="features", 
                        outputCol="scaled_features",
                        withStd=True,withMean=True)
scaler_model = scaler.fit(transformed_data)
scaled_data = scaler_model.transform(transformed_data)

Remember before you perform step 1, you need to assemble all the features with VectorAssembler. Hence this will be your step 0.

from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(inputCols=required_features, outputCol='features')
transformed_data = assembler.transform(df)