使用 PySpark 或 pandas_on_spark 计算新列

Question

I have this piece of code (below) that works perfectly on Pandas, but it is computationally too expensive to convert a large Dataframe to Pandas just for doing this operation. I'm looking for alternative solution how to do this in pandas-on-spark.

new_value = sum(df[col1]*df[col2])/sum(df[col2])

With pandas-on-spark I got following error: > PandasNotImplementedError: The method pd.Series.__iter__() is not implemented.

Answer 1

you can do it in spark Dataframe

from pyspark.sql.functions import sum as _sum
>>> df=df.withColumn("multiply",col("col1")*col("col2"))
>>> df.show(5)
+----+-----+--------+
|col1| col2|multiply|
+----+-----+--------+
| 001|   10|    10.0|
| 001|   10|    10.0|
| 002|   11|    22.0|
| 002|11878| 23756.0|
| 002|  117|   234.0|
+----+-----+--------+
only showing top 5 rows

>>> df.select(_sum('multiply')/_sum('col2')).show()
+---------------------------+
|(sum(multiply) / sum(col2))|
+---------------------------+
|          2.994753471275567|
+---------------------------+