编辑行并从文件中删除行

Question

I have a file of accession numbers and 16S rrna sequences, and what I'm trying to do is remove all lines of RNA, and only keep the lines with the accession numbers and the species name (and remove all the junk in between). So my input file looks like this (there are > in front of the accession numbers):

> D50541 1 1409 1409bp rna Abiotrophia defectiva Aerococcaceae
CUGGCGGCGU GCCUAAUACA UGCAAGUCGA ACCGAAGCAU CUUCGGAUGC UUAGUGGCGA ACGGGUGAGU AACACGUAGA UAACCUACCC UAGACUCGAG GAUAACUCCG GGAAACUGGA GCUAAUACUG GAUAGGAUAU AGAGAUAAUU UCUUUAUAUU (... and many more lines)

> AY538167 1 1411 1411bp rna Acholeplasma hippikon Acholeplasmataceae
CUGGCGGCGU GCCUAAUACA UGCAAGUCGA ACGCUCUAUA GCAAUAUAGG GAGUGGCGAA CGGGUGAGUA ACACGUAGAU AACCUACCCU UACUUCGAGG AUAACUUCGG GAAACUGGAG CUAAUACUGG AUAGGACAUA UUGAGGCAUC UUAAUAUGUU ...

I want my output to look like this:

>D50541 Abiotrophia defectiva Aerococcaceae

>AY538167 Acholeplasma hippikon Acholeplasmataceae

The code I wrote does what I want... for most of the lines. It looks like this:

    #!/usr/bin/env python

    # take LTPs111.compressed fasta and reduce to accession numbers with names.
    import re
    infilename = 'LTPs111.compressed.fasta'
    outfilename = 'acs.fasta'

    regex = re.compile(r'(>)\s(\w+).+[rna]\s+([A-Z].+)')    

    #remove extra letters and spaces
    with open(infilename, 'r') as infile, open(outfilename, 'w') as outfile:
        for line in infile:
            x = regex.sub(r'\1\2 \3', line)
    #remove rna sequences
        for line in x:
            if '>' in line:
                outfile.write(x)

Sometimes, the code seems to skip over some of the names. for example, for the first accession number above, I only got back:

>D50541 Aerococcaceae

Why might my code be doing this? The input for each accession number looks identical, and the spacing between 'rna' and the first name is the same for each line (5 spaces).

Thank you to anyone who might have some ideas!

Answer 1

I still haven't been able to run your code to get the claimed results, but I think I know what the problem is:

>>> line = '> AY538167 1 1411 1411bp rna Acholeplasma hippikon Acholeplasmataceae'
>>> regex = re.compile(r'(>)\s(\w+).+[rna]\s+([A-Z].+)')
>>> regex.findall(line)
[('>', 'AY538167', 'Acholeplasmataceae')]

The problem is that [rna]\\s+ matches any one of the characters r , n , or a at the end of a word. And, because all of the matches are greedy, with no lookahead or anything else to prevent it, this means that it matches the n at the end of hippikon .

The simple solution is to remove the brackets, so it matches the string rna :

>>> regex = re.compile(r'(>)\s(\w+).+rna\s+([A-Z].+)')

That won't work if any of your species or genera can end with that string. Are there any such names? If so, you need to come up with a better way to describe the cutoff between the 1409bp part and the rna part. The simplest may be to just look for rna surrounded by spaces:

>>> regex = re.compile(r'(>)\s(\w+).+\s+rna\s+([A-Z].+)')

Whether this is actually correct or not, I can't say without knowing more about the format, but hopefully you understand what I'm doing well enough to verify that it's correct (or at least to ask smarter questions than I can ask).

---

It may help debug things to add capture groups. For example, instead of this:

(>)\s(\w+).+[rna]\s+([A-Z].+)

… search for this:

(>)(\s)(\w+)(.+[rna]\s+)([A-Z].+)

Obviously your desired capture groups are now \\1\\3 \\5 instead of \\1\\2 \\3 … but the big thing is that you can see what got matched in \\4 :

[('>', ' ', 'AY538167', ' 1 1411 1411bp Acholeplasma hippikon ', 'Acholeplasmataceae')]

So, now the question is "Why did .+[rna]\\s+ match '1 1411 1411bp Acholeplasma hippikon ' ? Sometimes the context matters, but in this case, it doesn't. You don't want that group to match that string in any context, and yet it will always match it, so that's the part you have to debug.

---

Also, a visual regexp explorer often helps a lot. The best ones can color parts of the expression and the matched text, etc., to show you how and why the regexp is doing what it does.

Of course you're limited by those that run on your platform or online, and work with Python syntax. If you're careful and/or only use simple features (as in your example), perl/PCRE syntax is very close to Python, and JavaScript/ActionScript is also pretty close (the one big difference to keep in mind is that replace/sub uses $ instead of \\1 ).

I don't have a good online one to strongly recommend, but from a quick glance Debuggex looks pretty cool.

Answer 2

Items between brackets are character classes, so by setting your regex to look for "[rna]" you are requesting lines with either r, n, or a, but not all three.

Further, if the lines you want all have the pattern "bp rna", I'd use that to yank those lines. By reading the file in line by line, the following worked for me for a quick and dirty line-yanker, for instance:

regex = re.compile(r'^[\w\s]+bp rna .*$')

But, again, if it's as simple as finding lines with "bp rna" in them, you could read the file line by line and forego regex entirely:

for line in file:
   if "bp rna" in line:
     print(line) 

EDIT: I blew it by not reading the request carefully enough. Maybe a capture-and-replace regex would help?

for line in file:
  if "bp rna" in line:
    subreg = re.sub(r'^(>[\w]+)\s[\d\s]+bp\srna\s([\w\s]+$)', r"\1 \2", line)
    print(subreg)

OUTPUT:

>AY538166 Acholeplasma granularum Acholeplasmataceae

>AY538167 Acholeplasma hippikon Acholeplasmataceae

This should match any whitespace (tabs or spaces) between the things you want.