如何使用 Spark 3.0.0 读写 S3？

Question

I'm trying to launch a Spark application which should be able to read and write to S3, using Spark Operator on Kubernetes and pySpark version 3.0.0. The Spark Operator is wworking nicely, but I soon realized that the application launched can't read files from S3 properly.

This command:

spark.read.json("s3a://bucket/path/to/data.json")

is throwing this exception:

py4j.protocol.Py4JJavaError: An error occurred while calling o58.json.
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
Class org.apache.hadoop.fs.s3a.S3AFileSystem not found

I tried this first using gcr.io/spark-operator/spark-py:v3.0.0 as Spark image, and then tried adding some .jars to it with no success:

ADD https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.9/httpclient-4.5.9.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.7.4/hadoop-aws-2.7.4.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar $SPARK_HOME/jars

Here's my spark conf:

    "spark.hadoop.fs.s3a.endpoint": "S3A_ENDPOINT"
    "spark.hadoop.fs.s3a.access.key": "ACCESS_KEY"
    "spark.hadoop.fs.s3a.secret.key": "SECRET_KEY"
    "spark.hadoop.fs.s3a.connection.ssl.enabled": "false"
    "spark.hadoop.fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem"
    "spark.hadoop.fs.s3a.path.style.access": "true"
    "spark.driver.extraClassPath": "/opt/spark/jars/*"
    "spark.executor.extraClassPath": "/opt/spark/jars/*"

And my $SPARK_HOME is /opt/spark .

Is anyone able to read/write from S3 using Spark 3.0.0? Is this an issue with pyspark, exclusively? How can I "fix" this? Thanks in advance!

Answer 1

I figured out how to do it: Here's a fork with the changes I made to the base docker image (just a few changes):

https://github.com/Coqueiro/spark/tree/branch-3.0-s3 I created a Makefile to aid distribution creation, but I basically just followed the official doc:

http://spark.apache.org/docs/latest/building-spark.html

Also, here's the image, already built and pushed to Docker Hub: https://hub.docker.com/repository/docker/coqueirotree/spark-py

It has Spark 3.0.0, Hadoop 3.2.0, S3A and Kubernetes support.

Answer 2

Have you tried using spark jars that have pre-built Hadoop libraries https://spark.apache.org/downloads.html you can also add Hadoop dependencies to your classpath.