-c：第0行：意外令牌'（'附近的语法错误

Question

I have a PYTHON script which accepts a input file and gives an output file. the input file and output file are spceified as sys.argv[1] and [2] as below:

myscript.py input.file output.file

Unfortunately the input file needs to pre-processed to suit the requirements of the script. I can pre-process the raw files and convert into an acceptable format. The preprocessing step is as follow:

awk 'NR % 4 == 1 {print ">" $0 } NR % 4 == 2 {print $0}' filename.fastq > filename.fa

However, there are over 1000 such files and in a bid to save some storage space i want the script to take the pre-processed data through process substitution and i found a solution here: https://superuser.com/questions/1070265/how-to-pipe-awk-command-output-to-python-as-first-argument

So the following worked perfectly fine:

myscript.py <(awk 'NR % 4 == 1 {print ">" $0 } NR % 4 == 2 {print $0}' input.fastq) output_processed.fa

However, as i mentioned i have 1000 such files and i wrote a python script to loop over all the files and then nested the above command in os.system as follow

os.system("myscript.py <(awk 'NR % 4 == 1 {print ">" $0 } NR % 4 == 2 {print $0}' input.fastq) output_processed.fa")

However, this time when i run the script i get the following error:

sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `myscript.py <(awk 'NR % 4 == 1 {print ">" $0 } NR % 4 == 2 {print $0}' input.fastq) output_processed.fa'

i could find there are many answers for the same error, but none of them worked for me. i have some of the links below which i tried:

https://askubuntu.com/questions/348942/why-does-my-python-script-fail-with-syntax-errors

https://askubuntu.com/questions/656425/syntax-error-near-unexpected-token?rq=1

https://askubuntu.com/questions/656425/syntax-error-near-unexpected-token

https://askubuntu.com/questions/372926/bash-syntax-error-near-unexpected-token

when i used the above solution i got EOL error

I came to know that "(" is a wrong syntax from here : bash: syntax error near unexpected token `('

then i tried with "/" and then i get the following error:

 IOError: [Errno 2] No such file or directory: '/<(awk NR % 4 == 1 {print ">" $0 } NR % 4 == 2 {print $0} input.fastq)

Could you please suggest a solution for me?

Answer 1

Explicitly use bash for the process substitution

os.system("bash -c \"myscript.py <(awk -v gt='>' 'NR % 4 == 1 {print gt $0 } NR % 4 == 2 {print $0}' input.fastq) output_processed.fa\"")

I brought the ">" string out of the awk body to reduce the number of levels of nested quotes.

or use the default /bin/sh and write the awk output to a temp file.

os.system("awk 'NR % 4 == 1 {print \">\" $0 } NR % 4 == 2 {print $0}' input.fastq > temp && myscript.py temp output_processed.fa; rm -f temp")

If you control myscript.py , allow it to accept data on stdin instead of a filename, and you can write

os.system("awk '...' input.fastq | myscript.py output_processed.fa")