LINK : fatal error LNK1104: cannot open file 'C:\Users\hp\.pyxbld\lib.win32-2.7\gensim\models\word2vec_inner.pyd'

Question

I run the sourcecode of TWE model. I need to compile the C extension of python. I have installed the Microsoft Visual C++ Compiler for Python 2.7 and Cython.

First, I need to run the TWE/train.py:

import gensim
sentence_word = gensim.models.word2vec.LineSentence("tmp/word.file")
print "Training the word vector..."
w = gensim.models.Word2Vec(sentence_word,size=400, workers=20)
sentence = gensim.models.word2vec.CombinedSentence("tmp/word.file","tmp/topic.file")
print "Training the topic vector..."
w.train_topic(topic_number, sentence)
print "Saving the topic vectors..."
w.save_topic("output/topic_vector.txt")
print "Saving the word vectors..."
w.save_wordvector("output/word_vector.txt")`

Second, TWE/gensim/models/wor2vec.py:

try:
    raise ImportError  # ignore for now
    from gensim_addons.models.word2vec_inner import train_sentence_sg,train_sentence_cbow, FAST_VERSION, train_sentence_topic
except ImportError:
    try:
        import pyximport
        print 'import pyximport'
        models_dir = os.path.dirname(__file__) or os.getcwd()
        print 'models_dir'
        pyximport.install(setup_args={"include_dirs": [models_dir, get_include()]})
        print 'pyximport'   # is the follow code's problem
        from word2vec_inner import train_sentence_sg, train_sentence_cbow, 
        FAST_VERSION, train_sentence_topic
        print 'from word2vec'
    except:
        FAST_VERSION = -1
        def train_sentence_sg(model, sentence, alpha, work=None):
                   ...
        def train_sentence_cbow(model, sentence, alpha, work=None, neu1=None):
                   ...
class Word2Vec(utils.SaveLoad):
                   ...
    def train(self, sentences, total_words=None, word_count=0, chunksize=100):
        if FAST_VERSION < 0:
        import warnings
        warnings.warn("Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`")
        logger.info("training model with %i workers on %i vocabulary and %i features, "
        "using 'skipgram'=%s 'hierarchical softmax'=%s 'subsample'=%s and 'negative sampling'=%s" %
        (self.workers, len(self.vocab), self.layer1_size, self.sg, self.hs, self.sample, self.negative))
         def worker_train():
              ...
             if self.sg:
                 job_words = sum(train_sentence_topic(self, sentence, alpha, work) for sentence in job)
             else:
                 ob_words = sum(train_sentence_cbow(self, sentence, alpha, work, neu1) for sentence in job)`
              ...

Thrid, I haved compiled the TWE/gensim/models/word2vec_inner.pyx with a setup.py:

from distutils.core import setup  
from distutils.extension import Extension  
from Cython.Build import cythonize  
import numpy  
extensions = [  
    Extension("word2vec_inner", ["word2vec_inner.pyx"],  
              include_dirs=[numpy.get_include()])  
]  
setup(  
    name="word2vec_inner",  
    ext_modules=cythonize(extensions),  
)

by using the command 'python setup.py install', I have complied the word2vec_inner.pyx. But it appears the follow errors:

E:\Python27\python.exe D:/pycharm/TWE/TWE1/train.py wordmap.txt model-final.tassign 100
import pyximport
models_dir
pyximport
word2vec_inner.c
e:\python27\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg : Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
C:\Users\hp\.pyxbld\temp.win32-2.7\Release\pyrex\gensim\models\word2vec_inner.c(15079) : warning C4244:
'initializing' : conversion from 'double' to 'float', possible loss of data
C:\Users\hp\.pyxbld\temp.win32-2.7\Release\pyrex\gensim\models\word2vec_inner.c(15085) : warning C4244 : 'initializing' : conversion from 'double' to 'float', possible loss of data
LINK : fatal error LNK1104: cannot open file 'C:\Users\hp\.pyxbld\lib.win32-2.7\gensim\models\word2vec_inner.pyd'
Training the word vector...
D:\pycharm\TWE\TWE1\gensim\models\word2vec.py:410: UserWarning: Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`
warnings.warn("Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`")
PROGRESS: at 100.00% words, alpha 0.02500, 2556 words/s
Training the topic vector...
D:\pycharm\TWE\TWE1\gensim\models\word2vec.py:882: UserWarning: Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`
warnings.warn("Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`")
Exception in thread Thread-23:
Traceback (most recent call last):
  File "E:\Python27\lib\threading.py", line 801, in __bootstrap_inner
      self.run()
  File "E:\Python27\lib\threading.py", line 754, in run
      self.__target(*self.__args, **self.__kwargs)
  File "D:\pycharm\TWE\TWE1\gensim\models\word2vec.py", line 909, in worker_train
     job_words = sum(train_sentence_topic(self, sentence, alpha, work) for sentence in job)
  File "D:\pycharm\TWE\TWE1\gensim\models\word2vec.py", line 909, in <genexpr>
     job_words = sum(train_sentence_topic(self, sentence, alpha, work) for sentence in job)
NameError: global name 'train_sentence_topic' is not defined

Saving the topic vectors...
Saving the word vectors...

Process finished with exit code 0

I have checked that the .pyx file was compiled correctly and also installed the cython. in conclusion, it can't import train_sentence_sg,train_sentence_cbow, FAST_VERSION, train_sentence_topic from gensim/models/word2vec_inner or gensim_addons/models/word2vec_inner. So it appears these problems. But why? I have compiled the .pyx file correctly in both two directionarys. Anyone can help me? This problem haved troubled me several days. Please help me, thank you!

Answer 1

I ran into the same problem with PyCharm 2018.1 + Python 3.6.2.

This line is the key:

LINK : fatal error LNK1104: cannot open file 'C:\Users\hp\.pyxbld\lib.win32-2.7\gensim\models\word2vec_inner.pyd'

This error message is misleading. With Python, this error actually means:

• cannot open file to write to it .

The file is probably locked for writing by some process, so the linker cannot complete its job.

Solution 1

A previous Python line import word2vec_inner locked writing to the file. Reset the Python console to remove the lock:

Solution 2

Use Process Explorer to find out which program is locking the file. Use Ctrl-F, then type the name of the locked file in question, and it will give you the process that locked the file.

Solution 3

Exit Pycharm, then precompile the Cython file into a package on the command line . In case this link changes, here is a mirror:

Imagine a simple “hello world” script in a file hello.pyx:

 def say_hello_to(name): print("Hello %s!" % name) 

The following could be a corresponding setup.py script:

 from distutils.core import setup from Cython.Build import cythonize setup( name = 'Hello world app', ext_modules = cythonize("hello.pyx"), ) 

To build, run python setup.py build_ext --inplace . Then simply start a Python session and do from hello import say_hello_to and use the imported function as you see fit.

One caveat if you use setuptools instead of distutils, the default action when running python setup.py install is to create a zipped egg file which will not work with cimport for pxd files when you try to use them from a dependent package. To prevent this, include zip_safe=False in the arguments to setup().