如何使用 NLTK 从句子中获取单词对？

Question

I want to take in a sentence:

sentence = "How many people are here"?

and return a list of phrases:

pairs = ["How many", "many people", "people are", "are here"]

I tried

   tokens = nltk.word_tokenize(sentence)
   pairs = nltk.bigrams(tokens)

and instead got <generator object bigrams at 0x103697820>

Im pretty new to nltk so sorry that this is so off :) Help appreciated!

Answer 1

As you mentioned, the nktk.bigrams() function returns a generator object. Generators need to be iterated through in order to get the values out. This can be done with list() , or by looping over the generator.

Below, I'm looping/iterating over the generator object (results of nktk.bigrams() ) in a list comprehension, while at the same time using "".join() to combine the pair (list) of words, shed by the generator, into a single string, as desired.

tokens = nltk.word_tokenize(sentence)
pairs = [ " ".join(pair) for pair in nltk.bigrams(tokens)]

['How many', ...]

Answer 2

This should solve your problem:

import re
f = open('D:\Jupyter notebook\SNPQ.txt','r')
text = f.read()
text = re.sub('^\n|\n$','',(text))
for no,line in enumerate(text.splitlines()):
    print('"'+'","'.join([i.replace('"','\\"').strip() for i in re.split('(?<=^[0-9]{2})([0-9]{13}| {13})|  +',text.splitlines()[no].strip()) if i != None])+'"')

Thank you :)