[英]SyntaxError: invalid syntax when creating DataFrame with ArrayType
I want to create PySpark DataFrame我想创建 PySpark DataFrame
from pyspark.sql import SparkSession
from pyspark.sql.types import *
from pyspark.sql import Row
spark = SparkSession \
.builder \
.appName("Test") \
.master("local[4]") \
.getOrCreate()
schema = StructType([StructField('id', StringType()), \
StructField('timestamp',LongType()), \
StructField('coordinates',ArrayType())])
rows = [Row(id="11", timestamp=1523975430000, coordinates = [41.5555, 2.1522])]
df = spark.createDataFrame(rows, schema)
However, I get the syntax error SyntaxError: invalid syntax
next to lat
.但是,我在
lat
旁边收到语法错误SyntaxError: invalid syntax
。 I assume that ArrayType
objects should be created differently.我假设应该以不同的方式创建
ArrayType
对象。 Can someone help me creating this DataFrame df
?有人可以帮我创建这个 DataFrame
df
吗?
Update:更新:
Expected result:预期结果:
id timestamp coordinates
11 1523975430000 [41.5555, 2.1522]
ArrayType needs type of elements. ArrayType 需要元素类型。 Try:
尝试:
schema = StructType([StructField('id', StringType()), \
StructField('timestamp',LongType()), \
StructField('coordinates',ArrayType(DoubleType()))])
rows = [Row(id="11", timestamp=1523975430000, coordinates = [ 41.5555, 2.1522])]
Results:结果:
+---+-------------+-----------------+
| id| timestamp| coordinates|
+---+-------------+-----------------+
| 11|1523975430000|[41.5555, 2.1522]|
+---+-------------+-----------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.