How can I find the probability of a sentence using GPT-2?

Question

I'm trying to write a program that, given a list of sentences, returns the most probable one. I want to use GPT-2, but I am quite new to using it (as in I don't really know how to do it). I'm planning on finding the probability of a word given the previous words and multiplying all the probabilities together to get the overall probability of that sentence occurring, however I don't know how to find the probability of a word occurring given the previous words. This is my (psuedo) code:

sentences = # my list of sentences

max_prob = 0
best_sentence = sentences[0]

for sentence in sentences:
    prob = 1 #probability of that sentence

    for idx, word in enumerate(sentence.split()[1:]):
        prob *= probability(word, " ".join(sentence[:idx])) # this is where I need help

    if prob > max_prob:
        max_prob = prob
        best_sentence = sentence

print(best_sentence)

Can I have some help please?

Answer 1

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import numpy as np 


model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

def score(tokens_tensor):
    loss=model(tokens_tensor, labels=tokens_tensor)[0]
    return np.exp(loss.cpu().detach().numpy())

texts = ['i would like to thank you mr chairman', 'i would liking to thanks you mr chair in', 'thnks chair' ]
for text in texts:
    tokens_tensor = tokenizer.encode( text, add_special_tokens=False, return_tensors="pt")           
    print (text, score(tokens_tensor))

This code snippet could be an example of what are you looking for. You feed the model with a list of sentences, and it scores each whereas the lowest the better.

The output of the code above is:

i would like to thank you mr chairman 122.3066
i would liking to thanks you mr chair in 1183.7637
thnks chair 14135.129

Answer 2

I wrote a set of functions that can do precisely what you're looking for. Recall that GPT-2 parses its input into tokens (not words): the last word in 'Joe flicked the grasshopper' is actually three tokens: ' grass', 'ho', and 'pper'. The cloze_finalword function takes this into account, and computes the probabilities of all tokens (conditioned on the tokens appearing before them). You can adapt part of this function so that it returns what you're looking for. I hope you find the code useful!

Answer 3

You can also try lm-scorer, a tiny wrapper around transformers that allows you to get sentences probabilities using models that support it (only GPT2 models are implemented at the time of writing).

https://github.com/simonepri/lm-scorer

I just used it myself and works perfectly.

Answer 4

I think GPT-2 is a bit overkill for what you're trying to achieve. You can build a basic language model which will give you sentence probability using NLTK . A tutorial for this can be found here .