Python 未知模式查找

Question

Okay, basically what I want is to compress a file by reusing code and then at runtime replace missing code. What I've come up with is really ugly and slow, at least it works. The problem is that the file has no specific structure, for example 'aGVsbG8=\n', as you can see it's base64 encoding. My function is really slow because the length of the file is 1700+ and it checks for patterns 1 character at the time. Please help me with new better code or at least help me with optimizing what I got:). Anything that helps is welcome. BTW i have already tried compression libraries but they didn't compress as good as my ugly function.

def c_long(inp, cap=False, b=5):
    import re,string
    if cap is False: cap = len(inp)
    es = re.escape; le=len; ref = re.findall; ran = range; fi = string.find
    c = b;inpc = inp;pattern = inpc[:b]; l=[]
    rep = string.replace; ins = list.insert
    while True:
        if c == le(inpc) and le(inpc) > b+1: c = b; inpc = inpc[1:]; pattern = inpc[:b]
        elif le(inpc) <= b: break
        if c == cap: c = b; inpc = inpc[1:]; pattern = inpc[:b]
        p = ref(es(pattern),inp)
        pattern += inpc[c]
        if le(p) > 1 and le(pattern) >= b+1:
            if l == []: l = [[pattern,le(p)+le(pattern)]]
            elif le(ref(es(inpc[:c+2]),inp))+le(inpc[:c+2]) < le(p)+le(pattern):
                x = [pattern,le(p)+le(inpc[:c+1])]
                for i in ran(le(l)):
                    if x[1] >= l[i][1] and x[0][:-1] not in l[i][0]: ins(l,i,x); break
                    elif x[1] >= l[i][1] and x[0][:-1] in l[i][0]: l[i] = x; break
                inpc = inpc[:fi(inpc,x[0])] + inpc[le(x[0]):]
                pattern = inpc[:b]
                c = b-1
        c += 1
    d = {}; c = 0
    s = ran(le(l))
    for x in l: inp = rep(inp,x[0],'{%d}' % s[c]); d[str(s[c])] = x[0]; c += 1
    return [inp,d]

def decompress(inp,l): return apply(inp.format, [l[str(x)] for x in sorted([int(x) for x in l.keys()])])

Answer 1

The easiest way to compress base64-encoded data is to first convert it to binary data -- this will already save 25 percent of the storage space:

>>> s = "YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXo=\n"
>>> t = s.decode("base64")
>>> len(s)
37
>>> len(t)
26

In most cases, you can compress the string even further using some compression algorithm, like t.encode("bz2") or t.encode("zlib") .

A few remarks on your code: There are lots of factors that make the code hard to read: inconsistent spacing, overly long lines, meaningless variable names, unidiomatic code, etc. An example: Your decompress() function could be equivalently written as

def decompress(compressed_string, substitutions):
    subst_list = [substitutions[k] for k in sorted(substitutions, key=int)]
    return compressed_string.format(*subst_list)

Now it's already much more obvious what it does. You could go one step further: Why is substitutions a dictionary with the string keys "0" , "1" etc.? Not only is it strange to use strings instead of integers -- you don't need the keys at all, A simple list will do, and decompress() will simplify to

def decompress(compressed_string, substitutions):
    return compressed_string.format(*substitutions)

You might think all this is secondary, but if you make the rest of your code equally readable, you will find the bugs in your code yourself. (There are bugs -- it crashes for "abcdefgabcdefg" and many other strings.)

Answer 2

Typically one would pump the program through a compression algorithm optimized for text, then run that through exec , eg

code="""..."""
exec(somelib.decompress(code), globals=???, locals=???)

It may be the case that .pyc / .pyo files are compressed already, and one could check by creating one with x="""aaaaaaaa""" , then increasing the length to x="""aaaaaaaaaaaaaaaaaaaaaaa...aaaa""" and seeing if the size changes appreciably.