How to use image as input in tensorflow.js?

Question

I am working on using a convolutional neural network in a website that I am developing but am unsure how to create an input for an image.

The CNN model was trained in keras and then converted to a tensorflow.js format and is loading without problem. However, when I am trying to use an image as an input with the tf.fromPixels method within tensorflow.js, I am experiencing an issue stating:

"Uncaught DOMException: Failed to execute 'texImage2D' on 'WebGL2RenderingContext': The image element contains cross-origin data, and may not be loaded."

The image itself is locally stored and being shown in the webpage (the entire page is just being run locally at the moment). How would I go about feeding an image into the tensorflow.js CNN model? Is there anyways to use a local image through an html <img/> tag or does it have to be hosted online? My guess is that the fromPixels() method is causing a CORS error but I'm unsure if thats certainly the case and anyways around it.

Answer 1

You're right in that the CORS error is coming from WebGL's fromPixels() method.

There are two ways you can get around it.

First, You could request permission from the server with a function like

function requestCORSIfNotSameOrigin(img, url) {
  if ((new URL(url)).origin !== window.location.origin) {
    img.crossOrigin = ""; //this requests permission from the server
  }
}

and use it like this:

...
requestCORSIfNotSameOrigin(img, url);
img.src = url;

Note that some servers may not grant permission. Read more about WebGL CORS error here.

If you want to test your code without getting this error you could run a local server

$ python3 -m http.server [port]

and you won't get a CORS error.

Answer 2

My guess is that the fromPixels() method is causing a CORS error but I'm unsure if thats certainly the case and anyways around it.

Yes that's correct, so you need use a relative path and don't use file://

And regarding the <img> element if look at the documentation of .fromPixels() you can see it allows multiple types of pixel formats:

pixels ( ImageData|HTMLImageElement|HTMLCanvasElement|HTMLVideoElement ) The input image to construct the tensor from. The supported image types are all 4-channel.

One is HTMLImageElement so you can simply pass the <img> -element to .fromPixels() .