如何将 numpy 数组转换为 pyspark 数据框？

Question

I want to convert my results1 numpy array to a dataframe. For the record, results1 looks like

array([(1.0, 0.1738578587770462), (1.0, 0.33307021689414978),
       (1.0, 0.21377330869436264), (1.0, 0.443511435389518738),
       (1.0, 0.3278091162443161), (1.0, 0.041347454154491425)]).

I want to convert the above to a pyspark RDD with columns labeled "limit" (the first value in the tuple) and "probability" (the second value in the tuple).

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('YKP').getOrCreate()
sc=spark.sparkContext
# Convert list to RDD
rdd = sc.parallelize(results1)

# Create data frame
df = sc.createDataFrame(rdd)

I keep getting the error

AttributeError: 'RemoteContext' object has no attribute 'createDataFrame'

when I run this. I don't see why this is giving me an error and how do I fix this?

Answer 1

Use map() and toDF() instead.

import numpy as np

results1 = np.array([(1.0, 0.1738578587770462), (1.0, 0.33307021689414978),
       (1.0, 0.21377330869436264), (1.0, 0.443511435389518738),
       (1.0, 0.3278091162443161), (1.0, 0.041347454154491425)])

df = sc.parallelize(results1).map(lambda x: [float(i) for i in x])\
        .toDF(["limit", "probability"])

df.show()
+-----+--------------------+                                                    
|limit|         probability|
+-----+--------------------+
|  1.0|  0.1738578587770462|
|  1.0|  0.3330702168941498|
|  1.0| 0.21377330869436265|
|  1.0| 0.44351143538951876|
|  1.0|  0.3278091162443161|
|  1.0|0.041347454154491425|
+-----+--------------------+

Answer 2

The simplest way is:

df = rdd.map(lambda x: (x, )).toDF()
df.show()

You can also refer to this post for more details: Create Spark DataFrame. Can not infer schema for type: <type 'float'>