[英]Python: Compare regex pattern in one text file against line in another
This works in smaller text files, but not on larger. 这适用于较小的文本文件,但不适用于较大的文本文件 (100,000 lines) How can I optimize for large text files? (100,000行)如何优化大型文本文件? For line in fileA if regexPattern == line in fileB write (entire)line in fileA to fileC. 对于fileA中的行,如果regexPattern == fileB中的行将fileA中的(整个)行写入fileC。
import re
with open('fileC.txt', 'w') as outfile:
with open('fileA.txt', 'rU') as infile1:
for line1 in infile1:
y = re.findall(r'^.+,.+,(.+\.[a-z]+$)', line1)
with open('fileB.txt', 'rU') as infile2:
for line2 in infile2:
if line2.strip() == y[0]:
outfile.write(line1)
The most immediate optimization is to read fileB.txt
only once into a string buffer, then apply the test against the matched expression to that string buffer. 最直接的优化是只将fileB.txt
读入字符串缓冲区一次,然后将匹配表达式的测试应用于该字符串缓冲区。 You are currently opening and reading that file once for each line of fileA.txt
. 您当前正在为fileA.txt
每一行打开和读取该文件一次。
It seems that your regex picks up whole lines that match a pattern, ie it starts with ^
and ends with $
. 似乎你的正则表达式选择了匹配模式的整行,即它以^
开头并以$
结尾。 In this case, a more complete solution would be to load both fileA.txt
and fileB.txt
into arrays using readlines()
, sort those arrays, then take a single pass through both files with two counters, eg: 在这种情况下,更完整的解决方案是使用readlines()
将fileA.txt
和fileB.txt
到数组中,对这些数组进行排序,然后通过两个计数器单次传递两个文件,例如:
# Details regarding the treatment of duplicate lines are ignored
# for clarity of exposition.
rai = sorted([7,6,1,9,11,6])
raj = sorted([4,6,11,7])
i, j = 0, 0
while i < len(rai) and j < len(raj):
if rai[i] < raj[j]: i += 1
elif rai[i] > raj[j]: j += 1
else:
# I used mod in lieu of testing for your regex
# since you didnt supply data
if mod(rai[i],2): print rai[i]
i, j = i + 1, j + 1
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.