在文件Python中进行条件搜索和替换

Question

I have a large text file over 10MB when needs to have conditional search and replace. I want to replace every instance of "a" inside the file with "ā" if the character after "a" is either "r" or "m" or "n" or "u".

For example: Input file

Hamro sano ghar holata.

Output file

Hāmro sāno ghār holata.

EDIT

Thanks guys, it seems to work well. But it doesn't seem to work with non-latin characters like Indic Scripts: Working script for latin chars:

#!/usr/bin/env python
#-*- coding: utf-8 -*-
import re
input = "Hamro sano ghar holata."
regex = re.compile(ur'a([rmnu])')
print regex.sub(ur'ā\1', input)

Script1 (for Devanagari) NOT WORKING

#!/usr/bin/env python
#-*- coding: utf-8 -*-
import re
input ="संगम"
regex = re.compile(ur'ं([कखगघ])')
print regex.sub(r'ङ्\1', input)

Script2 (added unicode stuff) NOT WORKING

#!/usr/bin/env python
#-*- coding: utf-8 -*-
import re
input =u"संगम"
regex = re.compile(ur'ं([कखगघ])', re.UNICODE)
print regex.sub(r'ङ्\1', input)

Expected output: ं replaced with ङ् as ग follows ं that ie सङ्गम

Answer 1

you need a simple regular expression here. Something like this?

>>> import re
>>> input = "Hamro sano ghar holata."
>>> regex = re.compile(ur'a([rmnu])') # the part in parens is remembered
>>> print regex.sub(ur'ā\1', input) # replace by ā plus remembered part
Hāmro sāno ghār holata.

Edit:

some background, first:

This is a much tougher task with Devanāgarī (देवनागरी), not because of the encoding, but because the rules for combining the glyphs are extremely complicated (at least, by the standards of latin script). I'm writing this answer on Chrome, for example, which still can't compose the Devanāgarī for "Devanāgarī" correctly (it gets the diacritical mark for 'e' in the wrong place -- it does the same with the dipthong 'ai').

The ways these glyphs are combined by a text rendering engine are called 'ligatures', and for Devanāgarī they're very complicated, from a technical point of view. If you add the further enormous complications introduced by संधि ( saṃdhi -- again, Chrome's rendering gets the bindu that represents the anusvāra in the wrong place), then you can see that what you're trying to do here can quickly get extremely difficult.

Having said all that, if your problem is limited to this simple case, then I think it can be done cleanly.

>>> import re
>>> inputString = u"संगम"
>>> regex = re.compile(ur'\u0902(?=[कखगघ])')
>>> print regex.sub(ur'ङ\u094d', inputString)
सङ्गम

In the regexes I've replaced the anusvāra and the virāma (Hindi: halant ) with the unicode escaped value, for clarity. Given the way the ligatures work, it's possible this will miss some cases, but I've switched my example to using the lookahead, as in @Kabie's example (which is probably a better choice anyway), to mitigate this as far as possible.

Answer 2

re.sub(r'a(?=[rmnu])',r'ā',"Hamro sano ghar holata.")

Answer 3

For your large text file, you should copy the original, replace the characters, and write a new file with the updated lines. You should read just a chunk at a time, not the whole file. (Although on a modern computer you could just slurp the whole 10 MB in one go.)

An easy way to do this is to use the file object as an iterator; this returns one line from the file at a time.

import re
pat = re.compile(ur'a([rmnu])') # pre-compile regex pattern for speed

f = open("corrected_file.txt", "wb")

for line in open("big_file_10mb.txt", "rb"):
    line = pat.sub(ur'ā\1', line)
    f.write(line)

f.close()

If you wanted to slurp the whole file in one go, you can use the .read() method function:

f = open("big_file_10mb.txt", "rb")
s = f.read()  # read entire file contents
f.close()
s = pat.sub(ur'ā\1', s)  # replace over entire file contents
f = open("corrected_file.txt", "wb")
f.write(s)  # write entire file contents
f.close(s)

Don't do it this way unless you have a good reason. The line-oriented version is easy to understand and works much better when files are large compared to the memory available on your computer.

The book Dive Into Python has a chapter explaining regular expressions:

http://diveintopython3.ep.io/regular-expressions.html

You want to read Unicode and replace Unicode characters. You will need to figure out the native encoding of the file, read it in, convert to Unicode, do the substitution, then write it out in the proper encoding. Or you can use the special "codecs" module; codecs.open() will give you a file object that automatically converts for you.

Here is the Unicode "how-to" document for Python:

http://docs.python.org/howto/unicode.html

So, let's assume that the text file you want to read is encoded in UTF-8. I think this will work for you:

import codecs
import re

pat = re.compile(ur'a([rmnu])') # pre-compile regex pattern for speed

f = codecs.open("corrected_file.txt", mode="wb", encoding="utf-8")

for line in codecs.open("big_file_10mb.txt", mode="rb", encoding="utf-8"):
    line = pat.sub(ur'ā\1', line)
    f.write(line)

f.close()