I'm new in NLP. I'm trying to tokenize sentence using nlp on python 3.7.So I used following code
import nltk
text4="This is the first sentence.A gallon of milk in the U.S. cost
$2.99.Is this the third sentence?Yes,it is!"
x=nltk.sent_tokenize(text4)
x[0]
I was expecting that x[0] will return first sentence but I got
Out[4]: 'This is the first sentence.A gallon of milk in the U.S. cost $2.99.Is this the third sentence?Yes,it is!'
Am I doing anything wrong?
You need valid spacing and punctuation in your sentences for the tokenizer to behave properly:
import nltk
text4 = "This is a sentence. This is another sentence."
nltk.sent_tokenize(text4)
# ['This is a sentence.', 'This is another sentence.']
## Versus What you had before
nltk.sent_tokenize("This is a sentence.This is another sentence.")
# ['This is a sentence.This is another sentence.']
NLTK sent_tokenizer does not handle Ill formated text well. If you provide proper spacings then it works.
import nltk
nltk.download('punkt')
text4="This is the first sentence. A gallon of milk in the U.S. cost $2.99. Is this
the third sentence? Yes, it is"
x=nltk.sent_tokenize(text4)
x[0]
OR You could use this.
import re
text4 = "This is the first sentence. A gallon of milk in the U.S. cost 2.99. Is this
the third sentence? Yes it is"
sentences = re.split(r' *[\.\?!][\'"\)\]]* *', text4)
sentences
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.