Python-忽略Bigram频率中的数字和符号

Question

Im trying to find the Bi-gram frequency from a text from a txt files. So far it works but it counts numbers and symbols.Here is the code I have:

import  nltk
from nltk.collocations import *
import prettytable




file = open('tweets.txt').read()
tokens = nltk.word_tokenize(file)


pt = prettytable.PrettyTable(['Words', 'Counts'])
pt.align['Words'] = 'l'
pt.align['Counts'] = 'r'



bgs = nltk.bigrams(tokens)
fdist = nltk.FreqDist(bgs)

for row  in fdist.most_common(100):
    pt.add_row(row)
print pt


Below is the code output:
+------------------------------------+--------+
| Words                              | Counts |
+------------------------------------+--------+
| ('https', ':')                     |   1615 |
| ('!', '#')                         |    445 |
| ('Thank', 'you')                   |    386 |
| ('.', '``')                        |    358 |
| ('.', 'I')                         |    354 |
| ('.', 'Thank')                     |    337 |
| ('``', '@')                        |    320 |
| ('&', 'amp')                       |    290 |

Is there a way to ignore numbers and symbols ( like !,.,?,:)? Since the text are tweets, I want to ignore numbers and symbols, except for the #'s and @'s

Answer 1

An fdist for bigrams is a tuple of tuples containing a bigram tuple and a count integer, so we need to access the bigram tuple and keep only the ones we need in addition to the bigram's count. Try this:

import nltk
from nltk.probability import FreqDist
from nltk.util import ngrams
from pprint import pprint 

def filter_most_common_bigrams(mc_bigrams_counts):
    filtered_mc_bigrams_counts = []
    for mc_bigram_count in mc_bigrams_counts:
        bigram, count = mc_bigram_count
        #print (bigram, count)
        if all([gram.isalpha() for gram in bigram]) or bigram[0] in "#@" and bigram[1].isalpha():
            filtered_mc_bigrams_counts.append((bigram, count))
    return tuple(filtered_mc_bigrams_counts)   

text = """Is there a way to ignore numbers and symbols ( like !,.,?,:)?
Since the text are tweets, I want to ignore numbers and symbols, except for the #'s and @'s
https: !# . Thank you . `` 12 hi . 1st place 1 love 13 in @twitter # twitter"""

tokenized_text = nltk.word_tokenize(text)
bigrams = ngrams(tokenized_text, 2)
fdist = FreqDist(bigrams)
mc_bigrams_counts = fdist.most_common(100)     
pprint (filter_most_common_bigrams(mc_bigrams_counts))

The key piece of code is:

if all([gram.isalpha() for gram in bigram]) or bigram[0] in "#@" and bigram[1].isalpha():
    filtered_mc_bigrams_counts.append((bigram, count))

This checks that all the 1grams in the bigram are letters, or, alternatively, that the first bigram is a # or @ symbol and the second bigram is composed of letters. It only appends those that satisfy these conditions, and does so within a tuple containing the bigram's fdist count.

Results:

((('to', 'ignore'), 2),
 (('and', 'symbols'), 2),
 (('ignore', 'numbers'), 2),
 (('numbers', 'and'), 2),
 (('for', 'the'), 1),
 (('@', 'twitter'), 1),
 (('Is', 'there'), 1),
 (('text', 'are'), 1),
 (('a', 'way'), 1),
 (('Thank', 'you'), 1),
 (('want', 'to'), 1),
 (('Since', 'the'), 1),
 (('I', 'want'), 1),
 (('#', 'twitter'), 1),
 (('the', 'text'), 1),
 (('are', 'tweets'), 1),
 (('way', 'to'), 1),
 (('except', 'for'), 1),
 (('there', 'a'), 1))