从列表的子序列中删除重复项

Question

For part of log parser I need to filter occurrences of baud rate in the log.

First I get all occurrences using re.findall , then I'm trying to remove duplicates in subsequences in its result. Results are like [10000,10000,10000,10000,0,0,0,10000,10000] , the list can contain several hundreds of values. So the first baud rate was 10000 , then 0 , then again 10000 . I need to see how the baud rate changed, so I can't use set , as it will lose information of baud rate switching points.

So, once again input: [10000,10000,10000,10000,0,0,0,10000,10000]

Desired output: [10000,0,10000]

What I have made already:

m = [10000,10000,10000,10000,0,0,0,10000,10000] 
n = []
for i,v in enumerate(m):
    if i == 0:
        n.append(v)
        n_index = 0
    else:
        if v != n[n_index]:
            n.append(v)
            n_index = n_index + 1

it works, but it doesn't seem pythonic enough to me. Please advise: is there some more efficient way possible, or do I even not need to invent the wheel again?

Answer 1

Use itertools.groupby :

>>> rates = [10000,10000,10000,10000,0,0,0,10000,10000]
>>> from itertools import groupby
>>> [e for e, g in groupby(rates)]
[10000, 0, 10000]

Explanation: If no key function is given, then the elements are just grouped by identity, ie groups of consecutive equal elements are collapsed. The result is an iterator of key-elements and the groups (in this case, just repetitions of the key element). We need just the keys.

Update: Using IPython's %timeit magic command and a list of 100,000 random baud rates, itertools.groupby seems to be about as fast as the "compare to previous element loop" solutions, and a good deal shorter.

Answer 2

m = [10000,10000,10000,10000,0,0,0,10000,10000] 
n = []

n.append(m[0])
for i in m[1:]:
    if n[-1] != i:
        n.append(i)
print n

Answer 3

1. Iterate list m by normal method or by enumerate.
2. Check last element of list n is equal to current element of list m .
3. If not equal then append current element of list m to list 'n`.
4. Used try and expect because first time list n is empty

code :

m = [10000,10000,10000,10000,0,0,0,10000,10000] 
n = []
for v in m:
    try:
        if n[-1] != v:
            n.append(v)
    except IndexError:
        n.append(v)

print "Result:-", n 

Output:

$ python test.py 
Result:- [10000, 0, 10000]

Answer 4

I had to do the same thing, except I needed to save the position of each element. I am a physicist, so this code is probably crap, but it works. One can remove the aesthetic things like "print" and "press any key to continue..." as those were for debugging. Probably will adapt one of the answers on this thread.

xvalues=[]
M=[1,1,1,1,1,1,2,3,3,3,3,3,4,4,5,5,5,5,5,6,6,6,7,7,7,7,7,8,9,9]
print len(M)
i=0
while i < len(M):
    print "i " + str(i)
    j=i+1
    while j < len(M) and j > i:
        print "j " + str(j)
        if j == len(M)-1: #kills the while loop
            xvalues.append(i) #append the last element index
            print xvalues
            print range(i+1,len(M))
            a=raw_input("Press any key to continue...")
            i=len(M) #the loop killer
            break
        if M[i]!=M[j]:
            xvalues.append(i) #first index in the subsequence of duplicates
            print xvalues
            print range(i+1,len(M))
            a=raw_input("Press any key to continue...")
            i=j #skip to the next subsequence
            break
        if M[i]==M[j]:
            j+=1
            continue

Mnew=[M[i] for i in xvalues]
print xvalues
print Mnew

Final output. Position in array, then the value of that element.

[0, 6, 7, 12, 14, 19, 22, 27, 28]
[1, 2, 3,  4,  5,  6,  7,  8,  9]