如何仅生成二元/三元语料库

Question

Is there a way for Gensim to generate strictly the bigrams, trigrams in a list of words?

I can successfully generate the unigrams, bigrams, trigrams but I would like to extract only the bigrams, trigrams.

For example, in the list below:

words = [['the', 'mayor', 'of', 'new', 'york', 'was', 'there'],["i","love","new","york"],["new","york","is","great"]]

I use

bigram = gensim.models.Phrases(words, min_count=1, threshold=1)
bigram_mod = gensim.models.phrases.Phraser(bigram)
words_bigram = [bigram_mod[doc] for doc in words]

This creates a list of unigrams and bigrams as follows:

[['the', 'mayor', 'of', 'new_york', 'was', 'there'],
 ['i', 'love', 'new_york'],
 ['new_york', 'is', 'great']]

My question is, is there a way (other than regular expressions) to extract strictly the bigrams, so that in this example only "new_york" would be a result?

Answer 1

It's not a built-in option of the gensim Phrases functionality.

If we can assume none of your original unigrams had the '_' character in them, a step to select only tokens with a '_' shouldn't be too expensive (and doesn't need full regular expressions). For example, your last line could be:

words_bigram = [ [token for token in bigram_mod[doc] if '_' in token] for doc in words ]

(You could change the joining character if for some reason there were underscores in your unigrams, and you didn't want those confused with Phrases -combined bigrams.)

If none of that is good enough, you could potentially look at the code in gensim which actually scores & combines unigrams into bigrams...

https://github.com/RaRe-Technologies/gensim/blob/fbc7d0952f1461fb5de3f6423318ae33d87524e3/gensim/models/phrases.py#L300

...and either extend that module with your extra needed option, or mimic its behavior outside the class in your own code.