Windows UWP apps via Cordova: convert canvas into stream for InkRecognizer in Javascript

Question

I'm developing an application in Cordova for Android and Windows and struggle with the recogniztion of the text and numbers in canvas element on Windows platform (W10)

So last couple of days I've wasted my time trying to use the Windows.Media.OCR namespace for the recognition of the handwritten numbers on my HTML5 canvas scribble pad as you can see here on another SO question

I've then found the Windows.UI.Input.Inking namespace and there are few classes available for the Javascript solutions. I've found there is an InkManager that can recognize InkStrokes either in its own collection or strokes in InkRecognizerContainer.

InkRecognizerContainer has the "loadAsync()" method that accepts the input stream. So I've thought I'd just load the canvas converted to stream, and use the InkManager to recognize this container.

Unfortunately, if I try to use the HTML5 canvas converted to stream it throws me "WIN RT: Unsepcified Error" but not in the callbacks, it just crashes the app.

var blob = canvas.msToBlob();
 var randomAccessStream = blob.msDetachStream();

 var inkStrokeContainer = new Windows.UI.Input.Inking.InkStrokeContainer();
 inkStrokeContainer.loadAsync(randomAccessStream).done(function () {
     debugger
 }, function (error) {
    console.log(error);
 });

Any help would be greatly appreciated as I'm spending way too much time on this.

Answer 1

InkStrokeContainer.LoadAsync requires a file with ink stroke information, not an arbitrary bitmap. Generally this will be an ISF (Ink Serialized Format) file saved out from a previous InkStrokeContainer. ISF files include stroke information as metadata in a gif file, so they can be displayed by normal gif viewers, but typical gif files do not include ISF data and cannot load into InkStrokeContainers.

InkManager does handwriting recognition not OCR. It requires individual stroke information and takes into account properties such as stroke order and direction. To use it you'll need to pass pointer information to the InkManager, typically as the input occurs, so the InkManager can build the strokes to recognize.

Take a look at the Simplified Ink Sample for an example. The JavaScript version uses WinJS rather than Cordova, but it shouldn't be too hard to convert. The inking is Windows specific, so you'll need to put this in a platform specific part of your app.