简体繁体 English

Encog-如何为神经网络加载训练数据

[英]Encog - How to load training data for Neural Network

原文 2015-07-14 21:44:19 9 1 java/ machine-learning/ artificial-intelligence/ neural-network/ encog

The NeuralDataSet objects that I've seen in action haven't been anything but XOR which is just two small data arrays... I haven't been able to figure out anything from the documentation on MLDataSet . 我实际使用过的NeuralDataSet对象除了XOR之外什么都没什么，XOR只是两个小数据数组...我无法从MLDataSet的文档中找出任何东西。

It seems like everything must be loaded at once. 似乎必须立即加载所有内容。 However, I would like to loop through training data until I reach EOF and then count that as 1 epoch.. However, everything I've seen all the data must be loaded into 1 2D array from the beginning. 但是，我想遍历训练数据，直到到达EOF，然后将其计为1个时期。。但是，我所看到的所有数据都必须从一开始就加载到1个2D数组中。 How can I get around this? 我该如何解决？

I've read this question, and the answers didn't really help me . 我读过这个问题，答案并没有真正帮助我。 And besides that, I haven't found a similar question asked on here. 除此之外，我还没有在这里找到类似的问题。

1 个解决方案

This is possible, you can either use an existing implementation of a data set that supports streaming operation or you can implement your own on top of whatever source you have. 这是可能的，您可以使用支持流操作的数据集的现有实现，也可以在任何来源的基础上自己实现。 Check out the BasicMLDataSet interface and the SQLNeuralDataSet code as an example. 请以BasicMLDataSet接口和SQLNeuralDataSet代码为例。 You will have to implement a codec if you have a specific format. 如果您使用特定格式，则必须实施编解码器。 For CSV there is an implementation already, I haven't checked if it is memory based though. 对于CSV，已经有一个实现，但是我没有检查它是否基于内存。

Remember when doing this that your data will be streamed fully for each epoch and from my experience that is a much higher bottleneck than the actual computation of the network. 请记住，在执行此操作时，根据我的经验，您的数据将在每个时期被完全流式传输，这比网络的实际计算要高得多。