Spring boot - replacing properties in Hadoop core-site.xml config file

Question

I have Spring boot Spark based application with following Hadoop core-site.xml config file where I would like to replace the properties ${my.access.point.name} and ${my.access.point.arn} at runtime.

<configuration>
    <property>
        <name>fs.s3a.bucket.${my.access.point.name}.accesspoint.arn</name>
        <value>${my.access.point.arn}</value>
    </property>
</configuration>

Here I have specified both of this properties in application.properties file and also, tried to pass it as environment variables from the docker run command.

It isn't doing the replacement and end up getting Access Points usage is required but not configured for the bucket .

Is there any other way to replace this properties at runtime?

At the moment, this file core-site.xml sits under src/main/resources/ . Is there anyway I specify this file from external location as an alternative solution?

Answer 1

At runtime, you can get or create a custom Configuration instance.

For Spark, hadoopConfiguration is part of the SparkContext