如何使用 write() 方法使用 spark 在 s3 存储桶中写入 txt 文件

Question

I'm trying to write Dataset in txt format in s3 bucket using spark.

but I am getting the following error:

Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:64)
Caused by: java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified by setting the fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey properties (respectively).

enter image description here

My code:

override fun write(input: Dataset<String>) =

        input.coalesce(NUMBER_PARTITIONS).write().text(S3_BUCKET_PATH)
            .also {
                LOGGER.logInfo(
                    LOG_MESSAGE_TEMPLATE,
                    READ_DATA_METHOD,
                    WRITE_MESSAGE
                )
            }

enter image description here

My spark configuration: *

object SparkConfiguration {
    private const val SPARK_MASTER_NAME = "spark.master"
    private const val SPARK_APP_NAME_CONFIG = "spark.app.name"
    fun buildSparkSession(config: Config): SparkSession {
        return SparkSession.builder()
            .config(buildSparkConfig(config))
            .orCreate
    }
    fun buildSparkConfig(config: Config): SparkConf = SparkConf()
        .setMaster(config.getString(SPARK_MASTER_NAME))
        .setAppName(config.getString(SPARK_APP_NAME_CONFIG))
}

enter image description here

Answer 1

This is due to having no permissions when running your Spark Job:

Caused by: java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified by setting the fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey properties (respectively).

Make sure that you have permissions to run your code against S3.

Ideally set your credentials in the conf/core-site.xml as:

<configuration>
  <property>
    <name>fs.s3n.awsAccessKeyId</name>
    <value>XXXXXX</value>
  </property>

  <property>
    <name>fs.s3n.awsSecretAccessKey</name>
    <value>XXXXXX</value>
  </property>
</configuration>

or reinstall awscli on your machine and.

pip install awscli

then

aws configure