Best practice to create a live running model from a trained neural network

Question

I'm planning to create a neural network for trading, I'm investigating on how to use the trained neural network in a language that is used in runtime by my system (ie C++ or C#).

Let's say I'm using tensorflow in python (or any other nn framework), I've successfully trained my neural network that has n hidden layers. I have at that point the values for weights and bias. At this point I would like to use the nn model in production.

In my understanding it is not best practice to run in production tensorflow to run the model, since it's practically too slow (python) and clunky to do so (correct me if I'm wrong).

So I'd like to recreate the nn model in C# or C++ (or any other language), do I need to recreate in code all the maths for each layer based on weight and bias for every neuron?

Questions:

• I've never done it, so this is me guessing the workflow/best practice, what is the best way to do it?
• What are some online articles/websites I can read about this topic?
• What is this topic called, so that I can look for it?

Answer 1

Frameworks like tensorflow should also be used in production. Tensorflow is written in highly optimized C++ and/or Cuda and will thus easily outperform anything you will write yourself. There is also a large amount of inbuilt tools for monitoring, load balancing etc. that comes with tensorflow.

If you run your system on your machine you can use any language as a wrapper. If you really think you need a level of optimization for small tasks such as preprocessing etc. that can not be done in Python, just use the C++ API for tensorflow. Just let tensorflow (or torch or whatever production lvl framework you are using) do the heavy lifting of handling the neural network. For 99.99% percent of tasks Python(using numpy, tf, dask etc.) will be just fine.

For large scale applications you can host your model on aws/azure/google cloud etc. They offer optimized services for hosting tensorflow models (eg. as REST endpoints). Lambda fucntions/Azure Functions can be a great tool to do the preprocessing step in a costefficient and scaleable way.