reading config file in python with ConfigParser

Question

I have used configparser to read configurations in my python written program . I am reading file from s3 for now , but my requirement is to configurations defined in the program itself rather than from any other external source . code written is below:

config = configparser.ConfigParser()
config.readfp(open(s3:path\config))

config file format : config.ini

[section1]
var1=Y
var2=Y
var3=['col1','col2']

I am reading above file that is located in s3 , but instead of reading from s3 I want to read from program in itself . what needs to be done in order to achieve this ?

above code is written in pyspark program , I am passing config file with spark submit command but to read config file i need to provide path and this is not desirable . spark submit in aws emr :

'Args': ['spark-submit','--deploy-mode', 'cluster','--master', 'yarn','--executor-memory', conf['emr_step_executor_memory'],'--executor-cores',  conf['emr_step_executor_cores'],'--conf','spark.yarn.submit.waitAppCompletion=true','--conf','spark.rpc.message.maxSize=1024',f'{s3_path}/file1.py',  '--py-files',f'{s3_path}/file2.py',f'{s3_path}/file3.py',f'{s3_path}/file4.py','--files', f'{s3_path}/config ]

because of config.readfp(open(s3:path\\config)) line i need to provide s3 path , that is not desirable options are either pass config file from spark submit and make available to every other python files those are reading configs or read configuration inside of program itself .

Answer 1

you can do something like this:

parser = configparser.ConfigParser()
parser.read_dict({'section1': {'key1': 'value1',
                                'key2': 'value2',
                                'key3': 'value3'},
                   'section2': {'keyA': 'valueA',
                                'keyB': 'valueB',
                                'keyC': 'valueC'},
                   'section3': {'foo': 'x',
                                'bar': 'y',
                                'baz': 'z'}})

Check this for detailed info as well as some other alternative way to achieve this.

Sample retrieval

parser["section1"]["key1"]
'value1'