py4j.Py4JException：方法集（[class java.lang.String，class java.util.ArrayList]）在Databricks中不存在

Question

I am trying to access a remote S3 bucket using Databricks.

From what I have understood, this is what my code should look like,

sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", access_key)
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", secret_access_key)
sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "s3.amazonaws.com")


df = spark.read.format("csv").option("header",True).load('s3a://bucket/path/to/file.csv')

df.show()

I seem to be getting the following error when trying to set the Spark configurations,

Py4JError: An error occurred while calling o371.set. Trace:
py4j.Py4JException: Method set([class java.lang.String, class java.util.ArrayList]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:341)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:349)
    at py4j.Gateway.invoke(Gateway.java:286)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:251)
    at java.lang.Thread.run(Thread.java:750)

What am I doing wrong?

Answer 1

Check your configuration with S3 as per the official document :

Access_key = dbutils.secrets.get(scope = "Scope_name", key = "aws_access_key")
Secret_key = dbutils.secrets.get(scope = "Scope_name", key = "aws_secret_key")
sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", Access_key)
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", Secret_key)

region = "aws-region-id"
sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "s3." + region + ".amazonaws.com")

conn = sc.textFile("s3a://%s/.../..." % aws_bucket_name)
conn.count()

Or

You can try this alternative approach Mounting AWS S3 buckets on Databricks by Deepak Rajak