快速删除字符串中的所有非字母字符

Question

I have the following question. I am trying to remove all NON-letter characters from a string (meaning numbers (string.digits), punctuation marks (string.punctuation), non ascii characters (like φ,χ,ψ and so on). This can be done easily with a simple command like:

for i in str:
    if i not in string.ascii_letters: 
        data1 = str.replace(i,"")

or by using filter. However, my problem is that the length of my string is about 20.000.000 (several books concatenated together). Now in the case of 3.000.000 characters, the above command took about 20 minutes and therefore I did not dare to try it with 20.000.000 characters. Can you please tell me if there is any really really fast way to do that?

Answer 1

Something like that might eventually improve performances, as you don't have duplicate copy of your very long string in RAM:

data1 = (c for c in my_string if c in string.ascii_letters)

---

YMMV, but on my system, it takes something like 6s to filter a 20MB file containing random bytes (incl. the "".join(...) operation required to get back a string):

>>> data1 = (c for c in my_string if chr(ord(c)) in string.ascii_letters)
>>> timeit.timeit('"".join(data1)', setup='from __main__ import data1')
5.96341991424560

RegExp substitution took waaaayy much more time:

>>> timeit.timeit('re.sub("[^a-zA-Z]","",my_string)', setup='from __main__ import my_string; import re')
... still running after 90+ minutes...

Answer 2

我认为正则表达式就是为这种事情做的...

re.sub("[^a-zA-Z]","",my_string)