将数据帧写入镶木地板格式时出错

Question

I am trying to convert a dataframe to parquet format in S3 bucket of the AWS. But, i am getting the error that 's3a' bucket I am using is not found.I am using below code for the conversion.

df.write.mode('overwrite').parquet(folder_path)

The error I am getting is,

An error occurred while calling o328.parquet.
: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider not found
    at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:376)
    at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:51)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:229)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
    at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
    at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461)
    at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:556)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
    at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
    at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:781)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2571)
    at org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2642)
    at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:373)
    ... 26 more

I am using, spark 3.2.1 and Hadoop 3.2

I already downloaded the jar files, and pasted those in jar folder of spark, restarted my kernel.

please provide any solution.

Answer 1

1. IAMInstanceCredentialsProvider is in hadoop-aws 3.3.
2. in that version of hadoop the core-default.xml file in hadoop-common lists it as one of the default credential providers for fs.s3a.aws.credentials.provider.
3. As S3AUtils is found, there is some version of hadoop-aws on your classpath
4. But as it cannot find the file, it is of a version < 3.3.0

you have an inconsistent set of hadoop- jars, either locally or on the cluster. As the hadoop s3a troubleshooting docs note, mixing hadoop- jars is a way to see interesting stack traces, just as mixing spark- jars would be.

Fix: use a consistent set of hadoop jars across the entire cluster.