exclude some pattern in GNU parallel

Question

I want to use parallel to implement some files in one directory,

right now I have some tasks,

1, I want to skip some files, for example,

parallel -j 16 'zcat {} > {.}.unpacked' ::: *.gz

But for this I want to exclude some files with some pattern when operating this command. How can I implement this?

2, When some jobs when operating files are exited with error, how can I skip this status and continue operate other files?

Answer 1

You are a bit vague on what you want to exclude, but say you want to process all gzipped files except those starting with the letter a :

find -maxdepth 1 -iname "*.gz" ! -iname "a*" -print0 | parallel -0 'zcat {} > {.}.unpacked'

Regarding your second question, it is the default behaviour of GNU Parallel to continue after errors, so you shouldn't need to do anything explicitly. If you want to change it, look at the --halt option:

--halt now,fail=1 exit when the first job fails. Kill running jobs.

--halt soon,fail=3 exit when 3 jobs fail, but wait for running jobs to complete.

--halt soon,fail=3% exit when 3% of the jobs have failed, but wait for running jobs to complete.

--halt now,success=1 exit when a job succeeds. Kill running jobs.

--halt soon,success=3 exit when 3 jobs succeeds, but wait for running jobs to complete.

--halt now,success=3% exit when 3% of the jobs have succeeded. Kill running jobs.

--halt now,done=1 exit when one of the jobs finishes. Kill running jobs.

--halt soon,done=3 exit when 3 jobs finishes, but wait for running jobs to complete.

--halt now,done=3% exit when 3% of the jobs have finished. Kill running jobs.

Answer 2

If you do not want to use find , you can use skip() :

parallel -j 16 'zcat {= /mypattern/ and skip() =} > {.}.unpacked' ::: *.gz

/mypattern/ can be any Perl code.