How to load large dataset to gensim word2vec model

Question

So I have multiple text files(around 40). and each file has around 2000 articles (average of 500 words each). And each document is a single line in the text file.

So because of the memory limitations I wanted to use dynamic loading of these text files for training. (Perhaps a iterator class?)

so how do I proceed?

• train each text file -> save the model -> load the model and rerun on new data?
• is there a way with iterator class to do this automatically?
• should I give sentence by sentence, article by article or text file by text file as input to model training?

Answer 1

So I have multiple text files(around 40). and each file has around 2000 articles (average of 500 words each). And each document is a single line in the text file.

So because of the memory limitations I wanted to use dynamic loading of these text files for training. (Perhaps a iterator class?)

so how do I proceed?

• train each text file -> save the model -> load the model and rerun on new data?
• is there a way with iterator class to do this automatically?
• should I give sentence by sentence, article by article or text file by text file as input to model training?