如何在python中编写字母bigram（aa，ab，bc，cd…zz）频率分析计数器？

Question

This is my current code which prints out the frequency of each character in the input file.

from collections import defaultdict

counters = defaultdict(int)
with open("input.txt") as content_file:
   content = content_file.read()
   for char in content:
       counters[char] += 1

for letter in counters.keys():
    print letter, (round(counters[letter]*100.00/1234,3)) 

I want it to print the frequency of bigrams of only the alphabets(aa,ab,ac ..zy,zz) and not the punctuation as well. How to do this?

Answer 1

You can build around the current code to handle pairs as well. Keep track of 2 characters instead of just 1 by adding another variable, and use a check to eliminate non alphabets.

from collections import defaultdict

counters = defaultdict(int)
paired_counters = defaultdict(int)
with open("input.txt") as content_file:
   content = content_file.read()
   prev = '' #keeps track of last seen character
   for char in content:
       counters[char] += 1
       if prev and (prev+char).isalpha(): #checks for alphabets.
           paired_counters[prev+char] += 1
       prev = char #assign current char to prev variable for next iteration

for letter in counters.keys(): #you can iterate through both keys and value pairs from a dictionary instead using .items in python 3 or .iteritems in python 2.
    print letter, (round(counters[letter]*100.00/1234,3)) 

for pairs,values in paired_counters.iteritems(): #Use .items in python 3. Im guessing this is python2.
    print pairs, values

(disclaimer: i do not have python 2 on my system. if there is an issue in the code let me know.)

Answer 2

There is a more efficient way of counting bigraphs: with a Counter . Start by reading the text (assuming it is not too large):

from collections import Counter
with open("input.txt") as content_file:
   content = content_file.read()

Filter out non-letters:

letters = list(filter(str.isalpha, content))

You probably should convert all letters to the lower case, too, but it's up to you:

letters = letters.lower()    

Build a zip of the remaining letters with itself, shifted by one position, and count the bigraphs:

cntr = Counter(zip(letters, letters[1:]))

Normalize the dictionary:

total = len(cntr)
{''.join(k): v / total for k,v in cntr.most_common()}
#{'ow': 0.1111111111111111, 'He': 0.05555555555555555...}

The solution can be easily generalized to trigraphs, etc., by changing the counter:

cntr = Counter(zip(letters, letters[1:], letters[2:]))

Answer 3

If you're using nltk :

from nltk import ngrams
list(ngrams('hello', n=2))

[out]:

[('h', 'e'), ('e', 'l'), ('l', 'l'), ('l', 'o')]

To do a count:

from collections import Counter
Counter(list(ngrams('hello', n=2)))

If you want a python native solution, take a look at:

• Is there a more efficient way to find most common n-grams?
• Fast/Optimize N-gram implementations in python
• Effective 1-5 grams extraction with python