由于缓存奇怪问题，使用自己的数据集训练神经网络不起作用

Question

I faced a strange challenge trying to train neural network using code from github, it is huggingface conversational model.

What happens: even i use my own dataset for training result remains the same like with original dataset. My hypothesis that it is a somehow cache problem - old dataset continuously get loaded from cached and replace my. Them when i launch actual interactive session with neural network it works, but without my data, even if i pass model checkpoint.

Why i think of cache: in this repo author use automatic downloading and caching neural network model in /home/joo/.cache/torch/pytorch_transformers/ if no parameter specified in terminal.

I have created an issue on Github. BUT i am not sure is that a problem specific for this repo only, or it is a common problem with retraining neural networks i faced first time.

https://github.com/huggingface/transfer-learning-conv-ai/issues/36

Some copypaste from issue:

I am still curious, was not able to pass my dataset:

 I added to original 200mb json my personality trained once more with --dataset_path./my.json invoke interact.py with new checkpoint and path python./interact.py --model_checkpoint

./runs/Oct08_18-22-53_joo-tf_openai-gpt/ --dataset_path./my.json and it reports Gathered 18878 personalities (but not 18879, with my own). I changed the code in interact.py to choose my first perosnality this way

was: personality = random.choice(personalities)

become: personality = personalities[0]

and this first personality is not mine.

Answer 1

Solved: it is a specific issue to this repo, just hardcoded dataset path. But still why it doesn't load first time - no answer