如何加速在非常大的单单元 BAM 文件上使用正则表达式的 sed

Question

I have the following simple script that tries to count the tag encoded with "CB:Z" in SAM/BAM file :

samtools view -h small.bam |  grep "CB:Z:" |
    sed 's/.*CB:Z:\([ACGT]*\).*/\1/' |
    sort |
    uniq -c |
    awk '{print $2 " " $1}'

Typically it needs to process 40 million lines. That codes takes around 1 hour to finish.

This line sed 's/.*CB:Z:\([ACGT]*\).*/\1/' is very time consuming. How can I speed it up?

The reason I used the Regex is that the "CB" tag column-wise position is not fixed. Sometimes it's at column 20 and sometimes column 21.

Example BAM file can be found HERE .

---

Update

Speed comparison on complete 40 million lines file:

My initial code:

real    21m47.088s
user    26m51.148s
sys 1m27.912s

James Brown's with AWK:

real    1m28.898s
user    2m41.336s
sys 0m6.864s

James Brown's with MAWK:

real    1m10.642s
user    1m41.196s
sys 0m6.484s

Answer 1

Another awk, pretty much like @tripleee's, I'd assume:

$ samtools view -h small.bam | awk '
match($0,/CB:Z:[ACGT]*/) {               # use match for the regex match
    a[substr($0,RSTART+5,RLENGTH-5)]++   # len(CB:z:)==5, hence +-5
}
END {
    for(i in a)
        print i,a[i]                     # sample output,tweak it to your liking
}' 

Sample output:

...
TCTTAATCGTCC 175
GGGAAGGCCTAA 190
TCGGCCGATCGG 32
GACTTCCAAGCC 76
CCGCGGCATCGG 36
TAGCGATCGTGG 125
...

Notice : Your sed 's/.*CB:Z:... matches the last instance where as my awk 'match($0,/CB:Z:[ACGT]*/)... matches the first.

Notice 2 : Quoting @Sundeep in the comments: - - using LC_ALL=C mawk '..' will give even better speed.

Answer 2

With perl

perl -ne '$h{$&}++ if /CB:Z:\K[ACGT]++/; END{print "$_ $h{$_}\n" for keys %h}'

CB:Z:\K[ACGT]++ will match any sequence of ACGT characters preceded by CB:Z: . \K is used here to prevent CB:Z: from being part of matched portion, which is available via $& variable

Sample time with small.bam input file. mawk is fastest for this input, but it might change for larger input file.

# script.awk is the one mentioned in James Brown's answer
# result here shown with GNU awk
$ time LC_ALL=C awk -f script.awk small.bam > f1
real    0m0.092s

# mawk is faster compared to GNU awk for this use case
$ time LC_ALL=C mawk -f script.awk small.bam > f2
real    0m0.054s

$ time perl -ne '$h{$&}++ if /CB:Z:\K[ACGT]++/; END{print "$_ $h{$_}\n" for keys %h}' small.bam > f3
real    0m0.064s

$ diff -sq <(sort f1) <(sort f2)
Files /dev/fd/63 and /dev/fd/62 are identical
$ diff -sq <(sort f1) <(sort f3)
Files /dev/fd/63 and /dev/fd/62 are identical

Answer 3

Better to avoid parsing the output of samtools view in the first place. Here's one way to get what you need just using python and the pysam library:

import pysam

from collections import defaultdict

counts = defaultdict(int)
tag = 'CB'

with pysam.AlignmentFile('small.bam') as sam:
    for aln in sam:
        if aln.has_tag(tag):
            counts[ aln.get_tag(tag) ] += 1

for k, v in counts.items():
    print(k, v)

Answer 4

Following your original pipeline approach:

pcre2grep -o 'CB:Z:\K[^\t]*' small.bam |
 awk '{++c[$0]} END {for (i in c) print i,c[i]}'

In case you're interested in trying to speed up sed (although it's not likely to be the fastest):

sed 't a;s/CB:Z:/\n/;D;:a;s/\t/\n/;P;d' small.bam |
 awk '{++c[$0]} END {for (i in c) print i,c[i]}'

_{above syntax is compatible with GNU sed.}

Answer 5

regrading the AWK based solutions, i've noticed few taking advantage of FS.

I'm not too familiar with BAM format. If CB only show up once per line, then

mawk/mawk2/gawk -b 'BEGIN { FS = "CB:Z:"; 

   } $2 ~ /^[ACGT]/ {     # if FS never matches, $2 would be beyond
                          # end of line, then this would just match
                          # against  null string, & eval to false

      seen[substr($2, 1, -1 + match($2, /[^ACGT]|$/))]++ 

   } END { for (x in seen) { print seen[x] " " x } }' 

If it shows up more than once, then change that to a loop of any field greater than 1. This version uses the laziest evaluation model possible to speed it up, then do all the uniq -c item.

While this is rather similar to the best answer above, by having FS pre-split the fields, it causes match() and substr() to do a lot less work. I'm simply matching 1 single char after the genetic sequence, and directly using its return, minus 1, as the substring length, and skipping RSTART or RLENGTH all together.

Answer 6

Regarding:

$ diff -sq <(sort f1) <(sort f2)
Files /dev/fd/63 and /dev/fd/62 are identical
$ diff -sq <(sort f1) <(sort f3)
Files /dev/fd/63 and /dev/fd/62 are identical

there's absolutely no need to have them physically output to disk and do a diff. Just simply have the output of each piped to a very high speed hashing algorithm that adds close to no time (when the output is gigantic enough you might end up saving time versus going to disk.

my personal favorite is xxhash in 128-bit mode, available via python pip. it's NOT a cryptographic hash, but it's much faster than even something like MD5. This method also allows for hassle-free compare since the benchmark timing of it will also perform the accuracy check.