grep using a python (loop) variable and a string in a shell script

Question

Context: I have an internal Linux command, let's say "signals", which will give output whatever the signals placed for that day by oozie.

For example:

$signals | grep -i empty
*/user/oozie/coordinator/signals/2017/10/12/oltp1/_insurance_transaction_empty*
*/user/oozie/coordinator/signals/2017/10/12/oltp2/_loan_transaction_empty*

$signals | grep -i maxid
*/user/oozie/coordinator/signals/2017/10/12/oltp1/_insurance_transaction_maxid*
*/user/oozie/coordinator/signals/2017/10/12/oltp1/_loan_transaction_maxid*

I am writing a python script which runs every day and will grep for "empty" and "maxid" signals per oltp and send an email.

Let's say there are empty signal for oltp1 and maxid signal for oltp2. It will send an email with all oltps and signals.

I have a configfile which has all oltps in 1 column:

oltp1
oltp2
soltp
doltp

Code:

#!/usr/bin/python

import commands
import os

config_file="/home/xxx/config_file"
for var in confil_file:
    var=var.strip()
    print " checking with OLTP:"+var
    empty_cnt_loan=commands.getoutput("signals | grep -i empty | grep -i $var | wc -l")
    maxid_cnt_loan=commands.getoutput("signals | grep -i maxid | grep -i $var| wc -l")
    print empty_cnt_loan

attempts: tried with $var

ERROR: grep: write error:Broken Pipe

tried with %var empty_cnt_loan has value 0, where when i run the command in linux box, its has 2 value.

I tried subprocess, but since I am new I couldn't figure out how to use it.. Any help would be much appreciated.

Answer 1

Running a complex Bash pipeline to do things which are trivial in Python is an antipattern. Just run signals once and loop over the output in Python.

Assuming Python 3.6+ ,

#!/usr/bin/env python3

from subprocess import run, PIPE

with open("/home/xxx/config_file") as config:
    vars = [line.strip() for line in config.readlines()]
empty_cnt_loan = {x: 0 for x in vars}
maxid_cnt_loan = {x: 0 for x in vars}
p = run(['signals'], stdout=PIPE, check=True, universal_newlines=True)
for line in p.stdout.split('\n'):
    line = line.lower()
    for var in vars:
        if var in line:
            if 'maxid' in line:
                maxid_cnt_loan[var] += 1
            if 'empty' in line:
                empty_cnt_loan[var] += 1

As an aside, grep has an option -c to count the number of matches; but you really almost never need grep (or Awk, or sed , or cut , or a number of other standard utilities) when you have Python.