Reading .conf file from AWS s3 through spark and scala

Question

I was able to load a text file from AWS S3 but facing a problem in reading the ".conf" file. Getting the error

"Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'spark'"

Scala code:

val configFile1 = ConfigFactory.load( "s3n://<bucket_name>/aws.conf" )
configFile1.getString("spark.lineage.key")

Answer 1

Here what I end up doing it, Create a wrapper utility Config.scala

import java.io.File

import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
import com.amazonaws.services.s3.{AmazonS3Client, AmazonS3URI}
import com.typesafe.config.{ConfigFactory, Config => TConfig}

import scala.io.Source

object Config {

  private def read(location: String): String = {
    val awsCredentials = new DefaultAWSCredentialsProviderChain()
    val s3Client = new AmazonS3Client(awsCredentials)
    val s3Uri = new AmazonS3URI(location)

    val fullObject = s3Client.getObject(s3Uri.getBucket, s3Uri.getKey)

    Source.fromInputStream(fullObject.getObjectContent).getLines.mkString("\n")
  }

  def apply(location: String): TConfig = {

    if (location.startsWith("s3")) {
      val content = read(location)
      ConfigFactory.parseString(content)
    } else {
      ConfigFactory.parseFile(new File(location))
    }
  }
}

Use the created wrapper

val conf: TConfig = Config("s3://config/path")

You may use provided scope for aws-java-sdk since it will be available in the EMR cluster.

Answer 2

According to my research, we can only read delimiter files from AWS S3 through spark/scala. As .conf files are of = pair, its not possible.
Only way would be modify the format of data in the file.

Answer 3

类型安全配置不支持从 S3 加载 .conf 文件，但您可以自己将 s3 文件作为字符串读取并传递给类型安全配置，如val conf = ConfigFactory.parseString(... .conf files as string ...)