I want to create PySpark DataFrame
from pyspark.sql import SparkSession
from pyspark.sql.types import *
from pyspark.sql import Row
spark = SparkSession \
.builder \
.appName("Test") \
.master("local[4]") \
.getOrCreate()
schema = StructType([StructField('id', StringType()), \
StructField('timestamp',LongType()), \
StructField('coordinates',ArrayType())])
rows = [Row(id="11", timestamp=1523975430000, coordinates = [41.5555, 2.1522])]
df = spark.createDataFrame(rows, schema)
However, I get the syntax error SyntaxError: invalid syntax
next to lat
. I assume that ArrayType
objects should be created differently. Can someone help me creating this DataFrame df
?
Update:
Expected result:
id timestamp coordinates
11 1523975430000 [41.5555, 2.1522]
ArrayType needs type of elements. Try:
schema = StructType([StructField('id', StringType()), \
StructField('timestamp',LongType()), \
StructField('coordinates',ArrayType(DoubleType()))])
rows = [Row(id="11", timestamp=1523975430000, coordinates = [ 41.5555, 2.1522])]
Results:
+---+-------------+-----------------+
| id| timestamp| coordinates|
+---+-------------+-----------------+
| 11|1523975430000|[41.5555, 2.1522]|
+---+-------------+-----------------+
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.