python中的“ sed -f” bash命令等效

Question

I have a set of regular expressions for substitution in a file (sed.clean) as follow:

#!/bin/sed -f
s/https\?:\/\/[^ ]*//g
s/\.//g
s/\"//g
s/\,//g
y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/

and some more lines like those. I want to use this file for 'clean' a set of text files. To do this in bash I'd do something like this:

for file in $(ls rootDirectory)
do
    sed -f sed.clean $file > OUTPUT_FILE
done

How could I do something similar in Python?

What I mean is if it is possible to leverage the n RE which I have in the sed.clean file (or rewrite them in the proper Python format) in order to avoid building a nested loop to compare each file with each RE, and just compare each file with a sed.clean python file as I do in bash. Something like this:

files = [ f for f in listdir(dirPath) if isfile(join(dirPath,f)) ]
for file in files:
    newTextFile = re.sub(sed.clean, file)
    saveTextFile(newTextFile, outputPath)

instead of this:

REs = ['s/https\?:\/\/[^ ]*//g', 's/\.//g',...,'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/']
files = [ f for f in listdir(dirPath) if isfile(join(dirPath,f)) ]
for file in files:
    for re in REs:
        newTextFile = re.sub(re, '', file)
        saveTextFile(newTextFile, outputPath)

Thanks!

Answer 1

These sed patters appear to blank out lines matching certain patterns from a file. In python readlines() , filter() and re.sub() would be your best pick.

Answer 2

Try the re.sub like this:

import re
>>> re.compile(r'\.')
<_sre.SRE_Pattern object at 0x9d48c80>
>>> MY_RE = re.compile(r'\.')
>>> MY_RE.sub('','www.google.com')
'wwwgooglecom'

You can compile any regex in re.compile()

Answer 3

You'll have to convert your sed script replacements to Python equivalents.

---

s/<pattern>/<replacement>/<flags>
# is equivialent to
re.sub("<pattern>", "<replacement>", <input>, flags=<python-flags>)

Note that this is greedy, so there's no need for /g at the end of the pattern. Moreover, you should not include lags in the pattern, as they are passed as a separate parameter . For example:

re.sub("\.", "", "a.b.c.d", flags=re.MULTILINE)

---

y/<pattern>/<replacement>
# is equivivalent to
trans = str.maketrans("<pattern>", "<replacement>")
<input>.translate(trans)

But in the case of y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/ it's just as simple as <input>.lower() .

---

for file in $(ls rootDirectory) is roughly equivivalent to (taken from here )

files = [f for f in os.listdir('<rootDirectory>') if os.path.isfile(f)]
for f in files:
    # do something

---

All together:

import os # don't forget to import required modules
import re

output_file = open('C:\\temp\\output.txt', 'w')

def process(line):
    result = line
    result = re.sub("\"","", result)
    result = re.sub("\.","", result)
    # do all the stuff your sed script does and than
    return result

files = [f for f in os.listdir('.') if os.path.isfile(f)]
for file in files:
    file_handle = open(file_name, 'r')
    lines = file_handle.readlines()
    processed = map(process, lines)
    for line in processed:
        output_file.write(line)

Refer to the Python documentation for regex and file operations for details.

You might want to try converting your sed script to Python automatically, but if it's a one time requirement it's simpler to do it by hand.