按给定顺序检查列表的超集

Question

I have a list of tuples in format (float,string) sorted in descending order.

print sent_scores
[(0.10507038451969995,'Deadly stampede in Shanghai - Emergency personnel help victims.'),
 (0.078586381821416265,'Deadly stampede in Shanghai - Police and medical staff help injured people after the stampede.'),
 (0.072031446647399661, '- Emergency personnel help victims.')]

In case there two cases in the list which four words same in continuinty. I want to remove the tuple with lesser score from the list. The new list should also preserve order.

The output of above:

[(0.10507038451969995,'Deadly stampede in Shanghai - Emergency personnel help victims.')]

This will be first certainly involve tokenization of the words, which can be done the code below:

from nltk.tokenize import TreebankWordTokenizer

def tokenize_words(text):
    tokens = TreebankWordTokenizer().tokenize(text)
    contractions = ["n't", "'ll", "'m","'s"]
    fix = []
    for i in range(len(tokens)):
        for c in contractions:
            if tokens[i] == c: fix.append(i)
    fix_offset = 0
    for fix_id in fix:
        idx = fix_id - 1 - fix_offset
        tokens[idx] = tokens[idx] + tokens[idx+1]
        del tokens[idx+1]
        fix_offset += 1
    return tokens
 tokenized_sents=[tokenize_words(sentence) for score,sentence in sent_scores]

I earlier tried to convert the words of each sentences in groups of 4 contained a set and then use issuperset for other sentences. But it doesn't check continuity then.

Answer 1

I suggest taking sequences of 4 tokens in a row from your tokenized list, and making a set of those tokens. By using Python's itertools module, this can be done rather elegantly:

my_list = ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']
i1 = itertools.islice(my_list, 0, None)
i2 = itertools.islice(my_list, 1, None)
i3 = itertools.islice(my_list, 2, None)
i4 = itertools.islice(my_list, 3, None)
print zip(i1, i2, i3, i4)

Output of the above code (nicely formatted for you):

[('The', 'quick', 'brown', 'fox'),
 ('quick', 'brown', 'fox', 'jumps'),
 ('brown', 'fox', 'jumps', 'over'),
 ('fox', 'jumps', 'over', 'the'),
 ('jumps', 'over', 'the', 'lazy'),
 ('over', 'the', 'lazy', 'dog')]

Actually, even more elegant would be:

my_list = ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']
iterators = [itertools.islice(my_list, x, None) for x in range(4)]
print zip(*iterators)

Same output as before.

Now that you have your list of four consecutive tokens (as 4-tuples) for each list, you can stick those tokens in a set, and check whether the same 4-tuple appears in two different sets:

my_list = ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']
set1 = set(zip(*[itertools.islice(my_list, x, None) for x in range(4)]))

other_list = ['The', 'quick', 'red', 'fox', 'goes', 'home']
set2 = set(zip(*[itertools.islice(other_list, x, None) for x in range(4)]))

print set1.intersection(set2) # Empty set
if set1.intersection(set2):
    print "Found something in common"
else:
    print "Nothing in common"
# Prints "Nothing in common"

third_list = ['The', 'quick', 'brown', 'fox', 'goes', 'to', 'school']
set3 = set(zip(*[itertools.islice(third_list, x, None) for x in range(4)]))

print set1.intersection(set3) # Set containing ('The', 'quick', 'brown', 'fox')
if set1.intersection(set3):
    print "Found something in common"
else:
    print "Nothing in common"
# Prints "Found something in common"

NOTE : If you're using Python 3, just replace all the print "Something" statements with print("Something") : in Python 3, print became a function rather than a statement. But if you're using NLTK, I suspect you're using Python 2.

IMPORTANT NOTE : Any itertools.islice objects you create will iterate through their original list once , and then become "exhausted" (they've returned all their data, so putting them in a second for loop will produce nothing, and the for loop just won't do anything). If you want to iterate through the same list multiple times, create multiple iterators (as you see I did in my examples).

Update: Here's how to eliminate the lesser-scoring words. First, replace this line:

tokenized_sents=[tokenize_words(sentence) for score,sentence in sent_scores]

with:

tokenized_sents=[(score,tokenize_words(sentence)) for score,sentence in sent_scores]

Now what you have is a list of (score,sentence) tuples. Then we'll construct a list called scores_and_sets that will be a list of (score,sets_of_four_words) tuples (where sets_of_four_words is a list of four-word slices like in the example above):

scores_and_sentences_and_sets = [(score, sentence, set(zip(*[itertools.islice(sentence, x, None) for x in range(4)]))) for score,sentence in tokenized_sents]

That one-liner may be a bit too clever, actually, so let's unpack it to be a bit more readable:

scores_and_sentences_and_sets = []
for score, sentence in tokenized_sents:
    set_of_four_word_groups = set(zip(*[itertools.islice(sentence, x, None) for x in range(4)]))
    score_sentence_and_sets_tuple = (score, sentence, set_of_four_word_groups)
    scores_and_sentences_and_sets.append(score_sentence_and_sets_tuple)

Go ahead and experiment with those two code snippets, and you'll find that they do exactly the same thing.

Okay, so now we have a list of (score, sentence, set_of_four_word_groups) tuples. So we'll go through the list in order, and build up a result list consisting of ONLY the sentences we want to keep. Since the list is already sorted in descending order, that makes things a little easier, because it means that at any point in the list, we only have to look at the items that have already been "accepted" to see if any of them have a duplicate; if any of the accepted items are a duplicate of the one we've just looked at, we don't even need to look at the scores, because we know the accepted item came earlier than the one we're looking at, and therefore it must have a higher score than the one we're looking at.

So here's some code that should do what you want:

accepted_items = []
for current_tuple in scores_and_sentences_and_sets:
    score, sentence, set_of_four_words = current_tuple
    found = False
    for accepted_tuple in accepted_items:
        accepted_score, accepted_sentence, accepted_set = accepted_tuple
        if set_of_four_words.intersection(accepted_set):
            found = True
            break
    if not found:
        accepted_items.append(current_tuple)
print accepted_items # Prints a whole bunch of tuples
sentences_only = [sentence for score, sentence, word_set in accepted_items]
print sentences_only # Prints just the sentences