ML和DL4J阶段的示例

Question

I have a large S3 bucket full of photos of 4 different types of animals. My foray into ML will be to see if I can successfully get Deep Learning 4 Java (DL4J) to be shown a new arbitrary photo of one of those 4 species and get it to consistently, correctly guess which animal it is.

My understanding is that I must first perform a " training phase " which effectively builds up an (in-memory) neural network that consists of nodes and weights derived from both this S3 bucket (input data) and my own coding and usage of the DL4J library.

Once trained (meaning, once I have an in-memory neural net built up), then my understanding is that I can then enter zero or more " testing phases " where I give a single new image as input, let the program decide what type of animal it thinks the image is of, and then manually mark the output as being correct (the program guessed right) or incorrect w/ corrections (the program guessed wrong, and oh by the way, such and so was the correct answer). My understanding is that these test phases should help tweak you algorithms and minimize error.

Finally, it is my understanding that the library can then be used in a live " production phase " whereby the program is just responding to images as inputs and making decisions as to what it thinks they are.

All this to ask: is my understanding of ML and DL4J's basic methodology correction, or am I mislead in any way?

Answer 1

Training: That's any framework. You can also persist the neural network as well with either the java based SerializationUtils or in the newer release we have a ModelSerializer as well.

This is more of an integrations play than a "can it do x?"

DL4j can integrate with kafka/spark streaming and do online/mini batch learning.

The neural nets are embeddable in a production environment.

My only tip here is to ensure that you have the same data pipeline for training as well as test.

This is mainly for ensuring consistency of your data you are training vs testing on.

As well as for mini batch learning ensure you have minibatch(true) (default) if you are doing mini batch/online learning or minibatch(false) if you are training on the whole dataset at once.

I would also suggest using StandardScalar ( https://github.com/deeplearning4j/nd4j/blob/master/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/dataset/api/iterator/StandardScaler.java ) or something similar for persisting global statistics around your data. Much of the data pipeline will depend on the libraries you are using to build your data pipeline though.

I would assume you would want to normalize your data in some way though.