Count only the words in a text file Python

Question

I have to count all the words in a file and create a histogram of the words. I am using the following python code.

for word in re.split('[,. ]',f2.read()):
    if word not in histogram:
        histogram[word] = 1
    else:
        histogram[word]+=1

f2 is the file I am reading.I tried to parse the file by multiple delimiters but it still does not work. It counts all strings in the file and makes a histogram, but I only want words. I get results like this:

1-1-3:  3

where "1-1-3" is a string that occurs 3 times. How do I check so that only actual words are counted? casing does not matter. I also need to repeat this question but for two word sequences, so an output that looks like:

and the: 4

where "and the" is a two word sequence that appears 4 times. How would I group two word sequences together for counting?

Answer 1

from collections import Counter
from nltk.tokenize import RegexpTokenizer
from nltk import bigrams
from string import punctuation

# preparatory stuff
>>> tokenizer = RegexpTokenizer(r'[^\W\d]+')
>>> my_string = "this is my input string. 12345 1-2-3-4-5. this is my input"

# single words
>>> tokens = tokenizer.tokenize(my_string)
>>> Counter(tokens)
Counter({'this': 2, 'input': 2, 'is': 2, 'my': 2, 'string': 1})

# word pairs
>>> nltk_bigrams = bigrams(my_string.split())
>>> bigrams_list = [' '.join(x).strip(punctuation) for x in list(nltk_bigrams)]
>>> Counter([x for x in bigrams_list if x.replace(' ','').isalpha()])
Counter({'is my': 2, 'this is': 2, 'my input': 2, 'input string': 1})

Answer 2

Assuming you want to count all words in a string you could do something like this using a defaultdict as counters:

#!/usr/bin/env python3
# coding: utf-8

from collections import defaultdict

# For the sake of simplicty we are using a string instead of a read file
sentence = "The quick brown fox jumps over the lazy dog. THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG. The quick brown fox"

# Specify the word pairs you want to count as a single phrase
special_pairs = [('the', 'quick')]

# Convert sentence / input to lowercase in order to neglect case sensitivity and print lowercase sentence to double-check
sentence = sentence.lower()
print(sentence)


# Split string into single words
word_list = sentence.split(' ')
print(word_list)

# Since we know all the word in our input sentence we have to correct the word_list with our word pairs which need
# to be counted as a single phrase and not two single words
for pair in special_pairs:
    for index, word in enumerate(word_list):
        if pair[0] == word and pair[1] == word_list[index+1]:
            word_list.remove(pair[0])
            word_list.remove(pair[1])
            word_list.append(' '.join([pair[0], pair[1]]))


d = defaultdict(int)
for word in word_list:
    d[word] += 1

print(d.items())

Output:

the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox
['the', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog.', 'the', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog.', 'the', 'quick', 'brown', 'fox']
dict_items([('lazy', 2), ('dog.', 2), ('fox', 3), ('brown', 3), ('jumps', 2), ('the quick', 3), ('the', 2), ('over', 2)])