Unity - Running a Neural Network

Question

I trained a model using PyTorch. In Unity, I am using a WebCamTexture to display a live video. How can I feed the webcam frames into the PyTorch model, then perform actions in Unity with the output of the model?

I found Unity ML-agents, but it doesn't seem like it would help with this situation.

Answer 1

You can capture cam data on each update, then run your pytorch model by feeding it the data you just captured. I haven't tried and not sure how pytorch works, but for generic python scripts you can do something like:

...
void Start()
{
    ...
    data = new Color32[webcamTexture.width * webcamTexture.height];
    ...
}
...
void FixedUpdate ()
{
    ...
    webCamTexture.GetPixels32(data); //this is faster than returning a Color32 object
    ...
} 

...

private void runPython(string pathToPythonExecutable, string pyTorchScript, Color32[] data)
{
     var startInfo = new ProcessStartInfo();
     var pyTorchArgs = convertDataToYourPyTorchInputFormat (data)
     startInfo.Arguments = string.Format("{0} {1}", pyTorchScript, pyTorchArgs);
     startInfo.FileName = pathToPythonExecutable;
     startInfo.UseShellExecute = false;
     var process = Process.Start(start));
     process.WaitForExit();
     //do stuff in unity with the return value of process (process.ExitCode) or whatever.
}

Mind you, this may create significant overhead to create and end processes using an external executable file. There are some libraries that allow you to run python scripts inside c#. I can think of 2: IronPython ( http://ironpython.net ) and Python for .Net ( http://pythonnet.github.io ) I have never tried them though.