使用配置文件或CLI Args作为外部参数

Question

I am working on a Python project where the person who runs the program needs to specify some variables which then the program will use during execution. I would like to know to maintain production level coding standards what the preferred method is, use CLI or a JSONconfig file.

PS

To be more specific this is a Spark job and a developer who runs the job needs to specify these external parameters

Answer 1

A good method is to use a config.py file along with a .env file. This allows you to define variables psuedo-secretly in th .env file, and in config.py access them using python's dotenv and os libraries. The actual contents of the variables defined in .env will not be exposed in the project. For example, in the project directory you could have a file named .env containing lines like

# project/.env

foo='bar'
a=10
important_var='345v4325'

and in config.py ,

# project/config.py

import os
from dotenv import load_dotenv

load_dotenv()

secret_foo = os.environ.get('foo')
secret_a = os.environ.get('a')
secret_important_var = os.environ.get('important_var')

And then import config and use the vars like so

# project/module.py

import config

foo = config.secret_foo

So the project manager can modify things in .env without having to modify any code. If privacy is not an issue, you can skip the .env stuff and define variables directly in config.py

Notes
python-dotenv is not a standard python library. If you want to avoid new dependencies, similar functionality can be achieved by defining the variables in a bash script such as

# ~/.bashrc
export foo='bar'

# and so on...

and running source ~/.bashrc , then in config.py remove the from dotenv import load_dotenv and load_dotenv lines.

This works well if the code is being executed on a dedicated server where you can define you environment variables how you'd like.

Answer 2

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--someoption", type=str, help="something")
args = parser.parse_args()

print(args.someoption)

You can use the argparse package to pass the options in the CLI, pretty simple. You can read the document here

output :

vignesh@DESKTOP-C9L0NQI:/mnt/c/Users/vprabhakaran/Downloads/sunm$ ./argprs.py --someoption printthis
printthis

To parse a config file, you can use configparser package, very easy.