AWS EMR，EMR步骤中的python pyspark脚本

Question

I try to run a very simple pyspark script as a step in aws emr which looks like:

from pyspark.sql import SparkSession
sc = SparkContext()
df = sc.read.csv("s3://folder1/file.csv",header=True,inferSchema=True)
dd=df.select(df)
write_to = "s3://spark-workflow-test/"
dd.write.csv(write_to, sep = ";", header = True)
sc.stop()

It reads some file from a folder, selects a column, and writes it to another file in a bucket. For some reason it keeps failing and i cant figure out why.

This script works fine in local spark, but in an emr step it keeps failing and giving an exitCode=13. Is there are problem in the code, a spark configuration or do i need to do something in the console/emr infterface? I really have no clue about where to look for a solution.

Answer 1

I think your error is the same then in this issue.

Your spark context definition seems off. Replace it with :

sc = SparkSession.builder.getOrCreate()