python subprocess.check_output() ubuntu容器中json参数解析

Question

Trying to execute code with subprocess.check_output() inside an ubuntu 20.04 container:

try:
    out = subprocess.check_output(exe)
except subprocess.CalledProcessError as e:
    log.error(f"unable to execute:{' '.join(exe)} -> {e} -> {out}")
    return None

0211022170027|INFO|command used: ./bin/elkq.py --env=test --vertical=trk --index='bps-trace-*'
 --testenv=test12 --fields=@timestamp,message,bps.application --aggs=bps.application --start=now-4h
 --end=now --json --fany='("error")' --fmust='[{"fields.buypass.com/name": "id-pro-_VERT_-*"},
 {"bps.application": "*-_TESTENV_"}]'
20211022170028|ERROR|unable to execute:./bin/elkq.py --env=test --vertical=trk --index='bps-trace-*'
 --testenv=test12 --fields=@timestamp,message,bps.application --aggs=bps.application --start=now-4h
 --end=now --json --fany='("error")' --fmust='[{"fields.buypass.com/name": "id-pro-_VERT_-*"},
 {"bps.application": "*-_TESTENV_"}]' ->
 Command '['./bin/elkq.py', '--env=test', '--vertical=trk', "--index='bps-trace-*'",
 '--testenv=test12', '--fields=@timestamp,message,bps.application', '--aggs=bps.application',
 '--start=now-4h', '--end=now', '--json', '--fany=\'("error")\'',
 '--fmust=\'[{"fields.buypass.com/name": "id-pro-_VERT_-*"}, {"bps.application": "*-_TESTENV_"}]\'']'
 returned non-zero exit status 255. -> None

But if I cut and paste the command directly from the command use line above in the same container it works fine, so I assume the error is within the subprocess environment:

(base) root@deabf77921e8:/ # ./bin/elkq.py --env=test --vertical=trk --index='bps-trace-*' \
 --testenv=test12 --fields=@timestamp,message,bps.application --aggs=bps.application --start=now-4h \
 --end=now --json --fany='("error")' \
 --fmust='[{"fields.buypass.com/name": "id-pro-_VERT_-*"}, {"bps.application": "*-_TESTENV_"}]'
{
  "aggregations": {
    "bps.application": {
      "buckets": [
        {
          "doc_count": 15,
          "key": "id-pro-trk-trk-scim-api-test12"
        },
        {
          "doc_count": 1,
          "key": "id-pro-trk-trk-freg-gateway-test12"
        },
        {
          "doc_count": 1,
          "key": "id-pro-trk-trk-id-proofing-web-test12"
        }
      ],
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0
    }
  },
  "count": "max count found:17 rows in 0.05 sec"
}
(base) root@deabf77921e8:/ # echo $?
0

I know it has something to do with the --fmust parameter, because if I leave it out it works, and I earlier solved it by using a default parameter via argparse:

r'[{"fields.buypass.com/name": "id-pro-_VERT_-*"}, {"bps.application": "*-_TESTENV_"}]'

In powershell it works simply by double-quoting the " to "" but the CI runs linux containers.

Any help on debugging this very appreciated, thanks

Answer 1

I changed the subprocess.check_output() to use shell=True on linux, might be a dirty workaround but now it works in both powershell and bash with json parameters:

shell = True
if sys.platform != 'win32':
   exe = ' '.join(exe)
   shell = False
try:
    out = subprocess.check_output(exe, shell=shell)
except subprocess.CalledProcessError as e:
    log.error(f"unable to execute:{exe} -> {e} -> {out}")
    return None