使用Python，如何仅将不同的消息添加到列表？

Question

Certified Python Noob. Please bear with me.

In multiple files of a million (or so) lines of text, I'm going to filter out distinct ones. That is, even if of those millions contain nothing but 15 distinct lines, the code should return 15 lines.

Read line from file, put it in a list if it doesn't exist in there, output list to another file. Sounds simple?

There's a small thing though:

I'm looking for messages , not strings/substrings, or what have you. I'll explain below.

The Problem:

Suppose we have the following lines in the file:

• Random 2345
• Hello World
• Your code is 91939
• Your code is 54879
• Your code is 79865
• Pancakes 2451
• Your verification code is 123456
• Your verification code is 789101

Realistically, should I do a simple if line doesn't exist in myList, add line to myList , that would still return duplicates. The output should be:

• Random 2345
• Hello World
• Your code is 91939
• Pancakes 2451
• Your verification code is 123456

What I'm going to try:

Now, the numbers don't matter, so I'm might be able to get away with simply using regex or something to look for all numbers in the line, replace them with nothing, and then compare that to the list (which also had all numbers erased).

Crude, but it's the simplest I could think of.

Still with me?

More Problems:

Now comes the hard part. In addition to the above list, say we had the following:

• Hi, my name is Lance. Pleased to meet you.
• Hi, my name is Jenny. Pleased to meet you.
• Yo, dawg. Name's John Erik. Don't touch my fries.
• Yo, dawg. Name's James. Don't touch my fries.
• I like turtles 53669
• Stefan commented on your video.
• n00bpwn3rz commented on your video.
• RJ wants to talk to you.
• Jenny liked your photo.
• Pi wants to talk to you.
• Pi says visit my website at www.google.com
• John Erik says visit my website at www.johniscool.com
• James made fruity ice cubes.

And the output should be the following:

• Hi, my name is Lance. Pleased to meet you.
• Yo, dawg. Name's John Erik. Don't touch my fries.
• I like turtles 53669
• Stefan commented on your video.
• RJ wants to talk to you.
• Jenny liked your photo.
• Pi says visit my website at www.google.com
• James made fruity ice cubes.

My brain hurts. Not only do I have to take in names as variables, I have to watch out for websites too.

Now, suppose I dissect the line into Chars, loop-compare it to items from the list - also dissected into Chars - and if it hits X number of positives (Char from line = Char from list_item), I don't add it to the list. Is that feasible (as in, accurate)? How do I do that in code? Something like this, perhaps?

line_char[] = line       #My Name is Jayson
list_char[] = list_item  #My Name is Lance

if (list_char[] contains some sequence of line_char[]):
     #My Name is Jayson = My Name is Lance (12 TRUE [My Name is ], 6 FALSE [Lance/Jayson]; 12 > 6)
     line exists in list
else:
     add line to list

Any other ideas? This is probably more of a logic question, but I'd like to do this in Python, so I'll just take its advantages and limitations into account.

The Code so far:

Nothing to see here, folks.

import os

in_path = "../aggregator/"
out_path = "../aggregator_output/"
# For server: for filename in os.listdir(in_path):
# For local: for filename in list_path:
list_path = os.listdir(in_path)
del list_path[0]
for filename in list_path:
    in_base, in_ext = os.path.splitext(filename)
    in_file = os.path.join(in_path, filename)
    out_file = os.path.join(out_path, in_base + "_cleaned.csv")
    print "Processing " + in_file
    print "Writing to " + out_file
    dirty_file = open(in_file, "rb").read().split("\n")
    clean_file = open(out_file "wb")
    list_unique = []
    for line in dirty_file:
        temp_line = re.sub('",', '^', line)
        delimited = temp_line.split(",")
        message = delimited[2]

So far, all of my code is nothing more but to filter the right line from the file (3rd column).

I'd really appreciate some help on this, as this is a rather interesting problem, though one I can't solve myself.

Thanks.

PS - Commented out part of code pre-for-loop is to take in account that annoying .DS-store crap hidden file on a Mac, which breaks the rest of the code. I do testing on a Mac, and do the actual thing on an ubuntu server.

Answer 1

From what I see, you just want to keep the first different line when there are possible "duplicates" where you don't care about numbers or names...

Why not :

• Look at the first words of the line, if you find a new sequence of words you add it to the list
• Compare two strings representing two lines and define an interval from whom you know two lines are different.

For example, between these two lines :

• Hi, my name is Lance. Pleased to meet you.
• Hi, my name is Jenny. Pleased to meet you.

The only difference is Lance vs Jenny.

You can then code a compare function (because it doesn't exist in Python) based on the difference of the sum of ASCII code for all characters of the line. And say : two lines are similar if their "hash" is close.

Here is a sample of code for calculating the hash of a line :

class myString(str):
  def __hash__(self):
    count = 0
    for c in self:
      count += ord(c)
    return count

a = myString('Hi, my name is Lance. Pleased to meet you.')
b = myString('Hi, my name is Jenny. Pleased to meet you.')
c = myString("Yo, dawg. Name's John Erik. Don't touch my fries.")

hash(a) = 3624
hash(b) = 3657
hash(c) = 4148

Hope it will help ! Note that you can have problems with this solution with sentences that have the same sequence of characters, for example :

hash(myString('abc')) = 294
hash(myString('bac')) = 294

Answer 2

Since you are dealing with English sentences, I was wondering if nltk could be used for this. It provides a Parts of Speech (POS) tagger that can be used to find the POS in a sentence. Those lines with same sequence of tags are probably "similar" lines (This can also be improved further by comparing the actual tokens).

I tried it out for some example sentences from your question and looks like it's worth giving a try

import nltk

def pos_tags(text):
    return nltk.pos_tag(nltk.word_tokenize(text))

>>> pos_tags("Hi, my name is Lance. Pleased to meet you.")
[('Hi', 'NNP'),
(',', ','),
('my', 'PRP$'),
('name', 'NN'),
('is', 'VBZ'),
('Lance.', 'NNP'),
('Pleased', 'NNP'),
('to', 'TO'),
('meet', 'VB'),
('you', 'PRP'),
('.', '.')]

>>> pos_tags("Hi, my name is Jenny. Pleased to meet you")
[('Hi', 'NNP'),
(',', ','),
('my', 'PRP$'),
('name', 'NN'),
('is', 'VBZ'),
('Jenny.', 'NNP'),
('Pleased', 'NNP'),
('to', 'TO'),
('meet', 'VB'),
('you', 'PRP'),
('.', '.')]

The POS tags for each can then be encoded as strings and compared. If they are same there's a good chance that the lines are similar and can be grouped together.

>>> '-'.join([t[1] for t in pos_tags("Hi, my name is Jenny. Pleased to meet you")])
'NNP-,-PRP$-NN-VBZ-NNP-NNP-TO-VB-PRP-.'

>>> '-'.join([t[1] for t in pos_tags("Hi, my name is Lance. Pleased to meet you")])
'NNP-,-PRP$-NN-VBZ-NNP-NNP-TO-VB-PRP-.'

I am however not sure how it will perform on a million lines of text.

Answer 3

This approach is using a set and compare it with all known sets. If half of the words are in a given set it is assumed to be the same and skipped.

You have to give a clear definition when to sentences are similar so this can work.

a = """Hi, my name is Lance. Pleased to meet you.
Hi, my name is Jenny. Pleased to meet you.
Yo, dawg. Name's John Erik. Don't touch my fries.
Yo, dawg. Name's James. Don't touch my fries.
I like turtles 53669
Stefan commented on your video.
n00bpwn3rz commented on your video.
RJ wants to talk to you.
Jenny liked your photo.
Pi wants to talk to you.
Pi says visit my website at www.google.com
John Erik says visit my website at www.johniscool.com
James made fruity ice cubes."""


dirty_list = a.split('\n')
clean_list = [] # list of sets containing 'unique sets'
clean_list_pure = [] # list of the original sentences stored as sets in clean_list eg the output
for line in dirty_list:
    line_set = set(line.strip().split(' '))
    if all(len(line_set.intersection(clean_set)) < len(line_set)/2 for clean_set in clean_list):
        clean_list.append(line_set)
        clean_list_pure.append(line.strip())

for cl in clean_list_pure:
    print cl

as output here we get:

Hi, my name is Lance. Pleased to meet you.

Yo, dawg. Name's John Erik. Don't touch my fries.

I like turtles 53669

Stefan commented on your video.

Jenny liked your photo.

Pi says visit my website at www.google.com

James made fruity ice cubes.