How can I access the real-time sketch images on a p5.js canvas?

Question

Recently I am attempting to modify the source codes of this page . The underlying technique of this interactive programe is called sketch-rnn, a deep learning algorithm that can generate sketches. I need to access the real time images on the canvas so that I can use convolutional neural network (CNN), and feed the image as a 2d array to the neural network so that I can further improve the programe. Is there any p5.js function that can help me achieve that?

Answer 1

It depends in what format the CNN accepts input.

The simplest thing I can think of is using plain JavaScript (outside of p5.js) to access the <canvas /> element.

For example this is something you can try in your browser console on the sketch_rnn_demo page:

// access the default p5.js Canvas
canvasElement = document.querySelector('#defaultCanvas0')
// export the data as needed, for example encoded as a Base64 string:
canvasElement.toDataURL()

If you want to access pixels, you can via the Canvas context and getImageData() :

//access <canvas/> context
var context = canvasElement.getContext('2d');
//access pixels:
context.getImageData(0,0,canvasElement.width,canvasElement.height);

This will return a 1D array of unsigned 8-bit integers (eg values from 0-255) in R,G,B,A order (eg pixel0R,pixel0G,pixel0B,pixel0A,pixel1R,pixel1G,pixel1B,pixel1A...etc.)

If you want to use p5.js instead, call loadPixels() first, then access the pixels[] array which is the same format as above.

You can also use get(x,y) in p5.js which allows a 2D way to access to pixel data, however this is much slower.

If you CNN takes in a 2D array you still need to create this 2D array yourself and populate it pixel values (using pixels[] or get() for example). Be sure to double check the CNN input:

• it is a 2D array of 32-bit integers (eg R,G,B,A or A,R,G,B as a single int (0xAARRGGBB or 0xRRGGBBAA), just RGB, etc.)
• what resolution should the 2d array be? (your sketch-rnn canvas may be a different size and you might need to resize it to match what the CNN expects as an input)