“TypeError: join() argument must be str or bytes, not 'NoneType'” using Magpie+Tensorflow/Python3

Question

I am using magpie - a multilabel classifier on Tensorflow with Word2vec. The training works fine using Python 2.7, but with Python 3.x it gives this error below. I am on Ubuntu 16.04.5 LTS, weirdly the Python 3.x code works fine on a different Ubuntu machine. I have updated my dependencies: word2vec, magpie, tf and not sure what could be going wrong.

/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
Using TensorFlow backend.
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
Traceback (most recent call last):
  File "train_magpie.py", line 1, in <module>
    from magpie import Magpie
  File "/usr/local/lib/python3.5/dist-packages/magpie/__init__.py", line 1, in <module>
    from .main import Magpie
  File "/usr/local/lib/python3.5/dist-packages/magpie/main.py", line 12, in <module>
    from magpie.base.word2vec import train_word2vec, fit_scaler
  File "/usr/local/lib/python3.5/dist-packages/magpie/base/word2vec.py", line 7, in <module>
    from gensim.models import Word2Vec
  File "/usr/local/lib/python3.5/dist-packages/gensim/__init__.py", line 6, in <module>
    from gensim import parsing, matutils, interfaces, corpora, models, similarities, summarization
  File "/usr/local/lib/python3.5/dist-packages/gensim/parsing/__init__.py", line 7, in <module>
    from .preprocessing import *
  File "/usr/local/lib/python3.5/dist-packages/gensim/parsing/preprocessing.py", line 10, in <module>
    from gensim import utils
  File "/usr/local/lib/python3.5/dist-packages/gensim/utils.py", line 50, in <module>
    from smart_open import smart_open
  File "/usr/local/lib/python3.5/dist-packages/smart_open/__init__.py", line 1, in <module>
    from .smart_open_lib import *
  File "/usr/local/lib/python3.5/dist-packages/smart_open/smart_open_lib.py", line 45, in <module>
    from boto.compat import BytesIO, urlsplit, six
  File "/usr/lib/python3/dist-packages/boto/__init__.py", line 1216, in <module>
    boto.plugin.load_plugins(config)
  File "/usr/lib/python3/dist-packages/boto/plugin.py", line 92, in load_plugins
    for file in glob.glob(os.path.join(directory, '*.py')):
  File "/usr/lib/python3.5/posixpath.py", line 89, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/lib/python3.5/genericpath.py", line 143, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

Answer 1

The exception is coming from boto/plugin.py . For some reason, it's getting None instead of a string for some directory name, and then trying to glob all the files under None , which fails because you can't os.path.join a glob pattern to None .

So, why is it doing that?

Most likely, there's some garbage in your boto plugins directory. Or, if not there, in your config. (It would be nice if boto had better error handling for this case, but it probably doesn't come up all that often…)

There's a good chance just uninstalling and reinstalling your boto plugins will fix it.

---

For anyone reading this in the future:

If your exception ends with this (except possibly different line numbers and version numbers):

  File "/usr/lib/python3/dist-packages/boto/__init__.py", line 1216, in <module>
    boto.plugin.load_plugins(config)
  File "/usr/lib/python3/dist-packages/boto/plugin.py", line 92, in load_plugins
    for file in glob.glob(os.path.join(directory, '*.py')):
  File "/usr/lib/python3.5/posixpath.py", line 89, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/lib/python3.5/genericpath.py", line 143, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

… it's the same bug. But, if reinstalling your boto plugins doesn't help, you'll need to create a new question. (And include the results of calling find or tree on your plugins directory, and the contents of your config file, and all of the relevant version numbers.)