Why nltk word_tokenize is not working even after doing a nltk.download and all the packages are installed correctly?

Question

I am using python 3.7 64 bit. nltk version 3.4.5.

When I try to convert text6 in nltk.book to tokens using word_tokenize, I am getting error.

import nltk
from nltk.tokenize import word_tokenize
from nltk.book import *
tokens=word_tokenize(text6)

code is done in idle 3.7

Below is the error when I execute the last statement.

    Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    tokens=word_tokenize(text6)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\__init__.py", line 144, in word_tokenize
    sentences = [text] if preserve_line else sent_tokenize(text, language)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\__init__.py", line 106, in sent_tokenize
    return tokenizer.tokenize(text)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1277, in tokenize
    return list(self.sentences_from_text(text, realign_boundaries))
  File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1331, in sentences_from_text
    return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
  File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1331, in <listcomp>
    return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
  File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1321, in span_tokenize
    for sl in slices:
  File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1362, in _realign_boundaries
    for sl1, sl2 in _pair_iter(slices):
  File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 318, in _pair_iter
    prev = next(it)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1335, in _slices_from_text
    for match in self._lang_vars.period_context_re().finditer(text):
TypeError: expected string or bytes-like object

Please help. Thanks in advance.

While doing some troubleshooting I have created a sample nltk.text.Text object and tried to tokenize it with nltk.word_tokenize. Still I am getting the same error. Please see the below screenshot.

But while calling the nltk.word_tokenize() on string, its working.

>>> tt="Python is a programming language"
>>> tokens2=nltk.word_tokenize(tt) #Not throwing error
>>> type(tt)
<class 'str'>
>>> type(text6)
<class 'nltk.text.Text'>
>>> 

Answer 1

Check the nltk data folder. And where it expects it should be located.

Answer 2

Try using:

nltk.download('punkt')