如何有效地将Spark数据帧列转换为Numpy数组？

Question

I have a Spark dataframe with around 1 million rows. I am using pyspark and have to apply box-cox transformation from scipy library on each column of the dataframe. But the box-cox function allows only 1-d numpy array as input. How can I do this efficiently?

Is numpy array distributed on spark or it collects all the elements to single node on which driver program is running?

suppose df is my dataframe with column as C1 then, I want to perform the operation similar to this

stats.boxcox(df.select("C1"))

Answer 1

The dataframes/RDD in Spark allow abstracting from how the processing is distributed.

To do what you require, I think a UDF can be very useful. Here you can see an example of its use:

Functions from Python packages for udf() of Spark dataframe

Answer 2

I have a workaround that solve the issue but not sure is the optimal solution in term of performance as you are switching between pyspark and pandas dataframes:

dfpd = df.toPandas()
colName = 'YOUR_COLUMN_NAME'
colBCT_Name = colName + '_BCT'
print colBCT_Name
maxVal = dfpd[colName][dfpd[colName].idxmax()]
minVal = dfpd[colName][dfpd[colName].idxmin()]
print maxVal
print minVal

col_bct, l = stats.boxcox(dfpd[colName]- minVal +1)
col_bct = col_bct*l/((maxVal +1)**l-1)
col_bct =pd.Series(col_bct)
dfpd[colBCT_Name] = col_bct
df = sqlContext.createDataFrame(dfpd)
df.show(2)