Getting Error “stdbuf was not found; communication with perl may hang due to stdio buffering.” when using when using MosesTokenizer with Python

Question

problem arises when using mosestokenizer ( https://github.com/luismsgomes/mosestokenizer , installed through pypi https://pypi.org/project/mosestokenizer/ ) on windows platform (tried multiple windows machines).

Although there was no new releases during last month, before it used to work fine. I believe it has something to do with communication between python and perl.

>> MosesSentenceSplitter('en')
stdbuf was not found; communication with perl may hang due to stdio buffering.
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "...\.venv\lib\site-packages\mosestokenizer\sentsplitter.py",
    line 82, in __init__
        super().__init__(argv)
    File "...\.venv\lib\site-packages\toolwrapper.py",
    line 64, in __init__
        self.start()
    File "...\.venv\lib\site-packages\toolwrapper.py",
    line 102, in start
    self.proc = subprocess.Popen(
    File "C:\Users\MyUser\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py",
    line 854, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
    File "C:\Users\MyUser\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py",
    line 1307, in _execute_child
        hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

In order to reproduce it all it takes is to create some moses* class object:

from mosestokenizer import MosesSentenceSplitter
splitsents = MosesSentenceSplitter('en')  # and here error appear.

Answer 1

Installing perl fixed the error.

Also, as @ikegami mentioned, This should have $| = 1 $| = 1 , otherwise you will get stdbuf warnings.