设置Spark输出文件的S3输出文件被许可人

Question

I'm running Spark on AWS EMR and I'm having some issues getting the correct permissions on the output files ( rdd.saveAsTextFile('<file_dir_name>') ). In hive, I would add a line in the beginning with set fs.s3.canned.acl=BucketOwnerFullControl and that would set the correct permissions. For Spark, I tried running:

hadoop jar /mnt/var/lib/hadoop/steps/s-3HIRLHJJXV3SJ/script-runner.jar \
/home/hadoop/spark/bin/spark-submit --deploy-mode cluster --master yarn-cluster \
--conf "spark.driver.extraJavaOptions -Dfs.s3.canned.acl=BucketOwnerFullControl" \ 
hdfs:///user/hadoop/spark.py

But the permissions do not get set properly on the output files. What is the proper way to pass in the 'fs.s3.canned.acl=BucketOwnerFullControl' or any of the S3 canned permissions to the spark job?

Thanks in advance

Answer 1

I found the solution. In the job, you have to access the JavaSparkContext and from there get the Hadoop configuration and set the parameter there. For example:

sc._jsc.hadoopConfiguration().set('fs.s3.canned.acl','BucketOwnerFullControl')

Answer 2

The proper way to pass hadoop config keys in spark is to use --conf with keys prefixed with spark.hadoop. . Your command would look like

hadoop jar /mnt/var/lib/hadoop/steps/s-3HIRLHJJXV3SJ/script-runner.jar \
/home/hadoop/spark/bin/spark-submit --deploy-mode cluster --master yarn-cluster \
--conf "spark.hadoop.fs.s3.canned.acl=BucketOwnerFullControl" \ 
hdfs:///user/hadoop/spark.py

Unfortunately I cannot find any reference in official documentation of spark.