issues with sentence detection using nltk

Question

I have issues with nltk recognizing this as one sentence, because of the exclamation mark in the quotation marks.

s = "Donc ce n'est pas non plus de vous dire « Allez absolument ici ! », non."

I tried:

from nltk.tokenize import sent_tokenize
sent_tokenize(s, language='french')

but I get:

["Donc ce n'est pas non plus de vous dire « Allez absolument ici,", '». non.']

I am wondering if there is a better sentence detection method out there?

Answer 1

As someone commented below, you need it to handle other delimiters. Unfortunately, your example has an. which will automatically split irrespective of if you find a better tokeniser or not.

I have added another method that helps with multiple delimiters.

s = "Donc ce n'est pas non plus de vous dire « Allez absolument ici ! », non. hi there this is another sentence"

ss = s.split('.)
ss

["Donc ce n'est pas non plus de vous dire « Allez absolument ici ! », non",
 ' hi there this is another sentence']

Or you can use re.split for multiple delimiters

ss = re.split('[!.]',s)

["Donc ce n'est pas non plus de vous dire « Allez absolument ici ",
 ' », non',
 ' hi there this is another sentence']