NLTK punkt句子标记器在数字项目符号上拆分

Question

I am using nltk PunktSentenceTokenizer for splitting paragraphs into sentences. I have paragraphs as follows:

paragraphs = "1. Candidate is very poor in mathematics. 2. Interpersonal skills are good. 3. Very enthusiastic about social work"

Output: ['1.', 'Candidate is very poor in mathematics.', '2.', 'Interpersonal skills are good.', '3.', 'Very enthusiastic about social work']

I tried to add sent starters using below code but that didnt even work out.

from nltk.tokenize.punkt import PunktSentenceTokenizer
tokenizer = PunktSentenceTokenizer()
tokenizer._params.sent_starters.add('1.')

I really appreciate if anybody could drive me towards correct direction

Thanks in advance :)

Answer 1

The use of regular expressions can provide a solution to this type of problem, as illustrated by the code below:

paragraphs = "1. Candidate is very poor in mathematics. 2. Interpersonal skills are good. 3. Very enthusiastic about social work" 

import re
reSentenceEnd = re.compile("\.|$")
reAtLeastTwoLetters = re.compile("[a-zA-Z]{2}")

previousMatch = 0
sentenceStart = 0
end = len(paragraphs)
while(True):
    candidateSentenceEnd = reSentenceEnd.search(paragraphs, previousMatch)

    # A sentence must contain at least two consecutive letters:
    if reAtLeastTwoLetters.search(paragraphs[sentenceStart:candidateSentenceEnd.end()]) :
        print(paragraphs[sentenceStart:candidateSentenceEnd.end()])
        sentenceStart = candidateSentenceEnd.end()

    if candidateSentenceEnd.end() == end:
        break
    previousMatch=candidateSentenceEnd.start() + 1

the output is:

1. Candidate is very poor in mathematics.
2. Interpersonal skills are good.
3. Very enthusiastic about social work

Many tokenizers including (nltk and Spacy) can handle regular expressions. Adapting this code to their framework might not be trivial though.