如何将数据集转换为 glove 或 word2vec 格式？

Question

I have my twitter archive downloaded and wanted to run word2vec to experiment most similar words, analogies etc on it.

But I am stuck at first step - how to convert a given dataset / csv / document so that it can be input to word2vec? ie what is the process to convert data to glove/word2vec format?

Answer 1

Typically implementations of the word2vec & GLoVe algorithms do one or both of:

• accept a plain text file, where tokens are delimited by (one or more) spaces, and text is considered each newline-delimited line at a time (with lines that aren't "too long" - usually, short-article or paragraph or sentence per line)

• have some language/library-specific interface for feeding texts (lists-of-tokens) to the algorithm as a stream/iterable

The Python Gensim library offers both options for its Word2Vec class.

You should generally try working through one or more tutorials to get a working overview of the steps involved, from raw data to interesting results, before applying such libraries to your own data. And, by examining the formats used by those tutorials – and the extra steps they perform to massage the data into the formats needed by exactly the libraries you're using – you'll also see ideas for how your data needs to be prepared.