How to convert a image (Texture2D) to tensor

Question

I have c# TensorFlow.NET working in Unity. But it using an image from the file system. I want to be able to use an image from memory (Texture2D).

I tried to follow some examples of people using TensorFlowSharp. But that didn't work.

What am I doing wrong?

Note: with both functions, I am using the same image. The image is 512x512. but the result of both pictures is different.

// Doesn't work
private NDArray FromTextureToNDArray(Texture2D texture) {
    Color32[] pixels = texture.GetPixels32();

    byte[] floatValues = new byte[(texture.width * texture.height) * 3];

    for (int i = 0; i < pixels.Length; i++) {
        var color = pixels[i];

        floatValues[i * 3] = color.r;
        floatValues[i * 3 + 1] = color.g;
        floatValues[i * 3 + 2] = color.b;
    }

    Shape shape = new Shape(1, texture.width, texture.height, 3);
    NDArray image = new NDArray(floatValues, shape);

    return image;
}

// Works
private NDArray ReadFromFile(string fileName) {
    var graph = new Graph().as_default();

    // Change image
    var file_reader = tf.read_file(fileName, "file_reader");
    var decodeJpeg = tf.image.decode_jpeg(file_reader, channels: 3, name: "DecodeJpeg");

    var casted = tf.cast(decodeJpeg, TF_DataType.TF_UINT8);
    var dims_expander = tf.expand_dims(casted, 0);
    using (var sess = tf.Session(graph)) {
        return sess.run(dims_expander);
    }
}

Answer 1

I ended up using this code from Shaqian: https://github.com/shaqian/TF-Unity/blob/master/TensorFlow/Utils.cs

Add this script to your project and then you could use it like this:

// Get image
byte[] imageData = Utils.DecodeTexture(texture, texture.width, texture.height, 0, Flip.VERTICAL);
Shape shape = new Shape(1, texture.width, texture.height, 3);
NDArray image = new NDArray(imageData, shape);

Answer 2

Use Barracuda as step in between.

var encoder = new Unity.Barracuda.TextureAsTensorData(your_2d_texture);