wget and bash error: bash: line 0: fg: no job control

Question

I am trying to run a series of commands in parallel through xargs . I created a null-separated list of commands in a file cmd_list.txt and then attempted to run them in parallel with 6 threads as follows:

cat cmd_list.txt | xargs -0 -P 6 -I % bash -c %

However, I get the following error:

bash: line 0: fg: no job control

I've narrowed down the problem to be related to the length of the individual commands in the command list. Here's an example artificially-long command to download an image:

mkdir a-very-long-folder-de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8
wget --no-check-certificate --no-verbose -O a-very-long-folder-de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8/blah.jpg http://d4u3lqifjlxra.cloudfront.net/uploads/example/file/48/accordion.jpg

Just running the wget command on its own, without the file list and without xargs , works fine. However, running this command at the bash command prompt (again, without the file list) fails with the no job control error :

echo "wget --no-check-certificate --no-verbose -O a-very-long-folder-de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8/blah.jpg http://d4u3lqifjlxra.cloudfront.net/uploads/example/file/48/accordion.jpg" | xargs -I % bash -c %

If I leave out the long folder name and therefore shorten the command, it works fine:

echo "wget --no-check-certificate --no-verbose -O /tmp/blah.jpg http://d4u3lqifjlxra.cloudfront.net/uploads/example/file/48/accordion.jpg" | xargs -I % bash -c %

xargs has a -s (size) parameter that can change the max size of the command line length, but I tried increasing it to preposterous sizes (eg, 16000) without any effect. I thought that the problem may have been related to the length of the string passed in to bash -c , but the following command also works without trouble:

bash -c "wget --no-check-certificate --no-verbose -O a-very-long-folder-de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8/blah.jpg http://d4u3lqifjlxra.cloudfront.net/uploads/example/file/48/accordion.jpg"

I understand that there are other options to run commands in parallel, such as the parallel command ( https://stackoverflow.com/a/6497852/1410871 ), but I'm still very interested in fixing my setup or at least figuring out where it's going wrong.

I'm on Mac OS X 10.10.1 (Yosemite).

Answer 1

It looks like the solution is to avoid the -I parameter for xargs which, per the OS X xargs man page , has a 255-byte limit on the replacement string. Instead, the -J parameter is available, which does not have a 255-byte limit.

So my command would look like:

echo "wget --no-check-certificate --no-verbose -O a-very-long-folder-de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8/blah.jpg http://d4u3lqifjlxra.cloudfront.net/uploads/example/file/48/accordion.jpg" | xargs -J % bash -c %

However, in the above command, only the portion of the replacement string before the first whitespace is passed to bash, so bash tries to execute:

wget

which obviously results in an error. My solution is to ensure that xargs interprets the commands as null-delimited instead of whitespace-delimited using the -0 parameter, like so:

echo "wget --no-check-certificate --no-verbose -O a-very-long-folder-de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8de090952623b4865c2c34bd6330f8a423ed05ed8/blah.jpg http://d4u3lqifjlxra.cloudfront.net/uploads/example/file/48/accordion.jpg" | xargs -0 -J % bash -c %

and finally, this works!

Thank you to @CharlesDuffy who provided most of this insight. And no thank you to my OS X version of xargs for its poor handling of replacement strings that exceed the 255-byte limit.

Answer 2

I suspect it's the percent symbol, and your top shell complaining.

cat cmd_list.txt | xargs -0 -P 6 -I % bash -c %

Percent is a metacharacter for job control. "fg %2", eg "kill %4".

Try escaping the percents with a backslash to signal to the top shell that it should not try to interpret the percent, and xargs should be handed a literal percent character.

cat cmd_list.txt | xargs -0 -P 6 -I \\% bash -c \\%