unity ML Agents and external data

Question

Quite new to Unity, I'm more from a machine learning background. Planning to use ML Agents, and write some custom python / tensorflow scripts for it.

Is it possible to train based on data from my harddrive additionally to unity environment data? So for example having additional image data to feed as input to the network beside the Unity camera?

Haven't really seen that so far in the examples and documentation.

Thank you!

Answer 1

As long as you can express it in a fixed-length sequence of Vector3 , Vector2 , float , int , bool , Quaternion , or a fixed one-hot, sure no problem. Just include them with AddVectorObs in CollectObservations :

public override void CollectObservations()
{
    //internal info
    AddVectorObs(gameObject.transform.rotation.z);
    AddVectorObs(gameObject.transform.position);

    Vector3 externalInfo1 = ExternalInfoGetter.StaticGetInfo1();
    AddVectorObs(externalInfo1);

    float externalInfo2 = ExternalInfoGetter.StaticGetInfo2();
    AddVectorObs(externalInfo2);
}

See the documentation on designing agents for more info, including info on how to implement one-hot features and advice on normalizing the inputs.