Pyspark S3 NoClassDefFoundError: com/amazonaws/AmazonClientException

Question

I am trying to read S3 files from a small spark cluster I have running. I have the following jars installed:

"aws-java-sdk-bundle-1.11.975.jar"
"hadoop-aws-3.2.1.jar"

And am using the following code:

from pyspark.context import SparkContext
from pyspark.sql import SparkSession, SQLContext
import os

# initialise Spark session
spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.jars", "aws-java-sdk-bundle-1.11.975.jar") \
    .config("spark.jars", "hadoop-aws-3.2.1.jar") \
    .getOrCreate()

os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages com.amazonaws:aws-java-sdk-bundle-1.11.975,org.apache.hadoop:hadoop-aws-3.2.1 pyspark-shell'

fp = "s3a://filepath/objects/"
sc = spark.sparkContext
sqlContext = SQLContext(sc)
df = sqlContext.read.parquet(fp)

However when I run this, I get the error An error occurred while calling o62.parquet. : java.lang.NoClassDefFoundError: com/amazonaws/AmazonClientException An error occurred while calling o62.parquet. : java.lang.NoClassDefFoundError: com/amazonaws/AmazonClientException

If I update fp to be s3://... I get the error No FileSystem for scheme "s3"

I have tried a few solutions on here, but nothing seems to work so far.

Answer 1

Hadoop 3.2.1 was built with AWS SDK 1.11.375; could be a version issue there or simply the aws SDK JAR didn't get on the classpath

I'd start with the "troubleshooting s3a" page in the Hadoop docs