[英]Create labledpoints from Spark Dataframe & how to pass list of names to VectorAssembler
[英]How to create dataframe from list in Spark SQL?
Spark版本:2.1
例如,在pyspark中,我创建一个列表
test_list = [['Hello', 'world'], ['I', 'am', 'fine']]
那么如何从test_list创建一个数据框,其中数据框的类型如下所示:
DataFrame[words: array<string>]
这是怎么做的 -
from pyspark.sql.types import *
cSchema = StructType([StructField("WordList", ArrayType(StringType()))])
# notice extra square brackets around each element of list
test_list = [['Hello', 'world']], [['I', 'am', 'fine']]
df = spark.createDataFrame(test_list,schema=cSchema)
我不得不使用多个列和类型 - 下面的示例有一个字符串列和一个整数列。 对Pushkr代码的略微调整(上图)给出:
from pyspark.sql.types import *
cSchema = StructType([StructField("Words", StringType())\
,StructField("total", IntegerType())])
test_list = [['Hello', 1], ['I am fine', 3]]
df = spark.createDataFrame(test_list,schema=cSchema)
输出:
df.show()
+---------+-----+
| Words|total|
+---------+-----+
| Hello| 1|
|I am fine| 3|
+---------+-----+
您应该使用Row对象列表([Row])来创建数据框。
from pyspark.sql import Row
spark.createDataFrame(list(map(lambda x: Row(words=x), test_list)))
You can create a RDD first from the input and then convert to dataframe from the constructed RDD
<code>
import sqlContext.implicits._
val testList = Array(Array("Hello", "world"), Array("I", "am", "fine"))
// CREATE RDD
val testListRDD = sc.parallelize(testList)
val flatTestListRDD = testListRDD.flatMap(entry => entry)
// COnvert RDD to DF
val testListDF = flatTestListRDD.toDF
testListDF.show
</code>
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.