使用映射重命名gensim Word2Vec单词

Question

I want to replace the words of my gensim Word2Vec model with a mapping.

Example

My current model has the word 'foo' that maps to a vector:

>>> model['foo']
[1.0 0.0]

I have the mapping: d = {'foo': 'bar', ...}

How can I rebuild the model with this new mapping such that

>>> model['bar']  # in place of 'foo'
[1.0 0.0]

Answer 1

One solution is to save the model in the C-based word2vec format and replace the original words with a mapping of the new words using awk .

Assume we have a file mapping of the form:

$ cat map.txt
foo:bar
...

We can recreate the model via:

import subprocess as sp
import shlex

from gensim.models import Word2Vec

model.save_word2vec_format('embeddings.txt', binary=False)

CMD = r"""
awk -F'[ ]|:' 'FNR==NR {a[$1]=$2; next} FNR==1{print $0} FNR!=1{$1=a[$1]; print $0}' map.txt embeddings.txt
"""

with open('new_embeddings.txt', 'w') as f:
    p = sp.Popen(shlex.split(CMD), stdout=f)

new_model = Word2Vec.load_word2vec_format('new_embeddings.txt')

new_model.create_binary_tree()

As an aside my mapping was actually an array where I was training on the index of the word in some array arr . I created the map file using numpy:

import numpy as np

np.savetxt('map.txt', np.c_[np.arange(arr.size), arr], '%d:%s')