AWS Lambda w/ Google Vision API throwing PEM_read_bio:no start line or Errno::ENAMETOOLONG

Question

The Goal: User uploads to S3, Lambda is triggered to take the file and send to Google Vision API for analysis, returning the results.

According to this , google-cloud requires native libraries and must be compiled against the OS that lambda is running. Using lambda-packager threw an error but some internet searching turned up using an EC2 with Node and NPM to run the install instead. In the spirit of hacking through this, that's what I did to get it mostly working*. At least lambda stopped giving me ELF header errors.

My current problem is that there are 2 ways to call the Vision API, neither work and both return a different error (mostly).

---

The Common Code: This code is always the same, it's at the top of the function, and I'm separating it to keep the later code blocks focused on the issue.

'use strict';

const AWS = require('aws-sdk');
const S3 = new AWS.S3();
const Bucket = 'my-awesome-bucket'; 

const gCloudConfig = {
    projectId: 'myCoolApp',
    credentials: {
        client_email: 'your.serviceapi@project.email.com',
        private_key: 'yourServiceApiPrivateKey'
    }
}
const gCloud = require('google-cloud')(gCloudConfig);
const gVision = gCloud.vision();

---

Using detect() : This code always returns the error Error: error:0906D06C:PEM routines:PEM_read_bio:no start line . Theoretically it should work because the URL is public. From searching on the error, I considered it might be an HTTPS thing, so I've even tried a variation on this where I replaced HTTPS with HTTP but got the same error.

exports.handler = (event, context, callback) => {
    const params = {
        Bucket,
        Key: event.Records[0].s3.object.key
    }
    const img = S3.getSignedUrl('getObject', params);
    gVision.detect(img, ['labels','text'], function(err, image){
        if(err){
            console.log('vision error', err);
        }
        console.log('vision result:', JSON.stringify(image, true, 2));
    });
}

---

Using detectLabels() : This code always returns Error: ENAMETOOLONG: name too long, open ....[the image in base64]... . On a suggestion, it was believed that the method shouldn't be passed the base64 image, but instead the public path; which would explain why it says the name is too long (a base64 image is quite the URL). Unfortunately, that gives the PEM error from above. I've also tried not doing the base64 encoding and pass the object buffer directly from aws but that resulted in a PEM error too.

exports.handler = (event, context, callback) => {
    const params = {
        Bucket,
        Key: event.Records[0].s3.object.key
    }
    S3.getObject(params, function(err, data){
        const img = data.Body.toString('base64');
        gVision.detectLabels(img, function(err, labels){
            if(err){
                console.log('vision error', err);
            }
            console.log('vision result:', labels);
        });
    });
}

---

According to Best Practices , the image should be base64 encoded.

From the API docs and examples and whatever else, it seems that I'm using these correctly. I feel like I've read all those docs a million times.

I'm not sure what to make of the NAMETOOLONG error if it's expecting base64 stuff. These images aren't more than 1MB.

*The PEM error seems to be related to credentials, and because my understanding of how all these credentials work and how the modules are being compiled on EC2 (which doesn't have any kind of PEM files), that might be my problem. Maybe I need to set up some credentials before running npm install , kind of in the same vein as needing to be installed on a linux box? This is starting to be outside my range of understanding so I'm hoping someone here knows.

Ideally, using detect would be better because I can specify what I want detected, but just getting any valid response from Google would be awesome. Any clues you all can provide would be greatly appreciated.

Answer 1

So, a conversation with another colleague pointed me to consider abandoning the whole loading of the API and using the google-cloud module. Instead, I should consider trying the Cloud REST API via curl and seeing if it can work that way.

Long story short, making an HTTP request and using the REST API for Google Cloud was how I solved this issue.

Here is the working lambda function I have now. Probably still needs tweaks but this is working.

'use strict';

const AWS = require('aws-sdk');
const S3 = new AWS.S3();
const Bucket = 'yourBucket';
const fs = require('fs');
const https = require('https');

const APIKey = 'AIza...your.api.key...kIVc';

const options = {
    method: 'POST',
    host: `vision.googleapis.com`,
    path: `/v1/images:annotate?key=${APIKey}`,
    headers: {
        'Content-Type': 'application/json'
    }
}

exports.handler = (event, context, callback) => {
    const req = https.request(options, res => {
        const body = [];
        res.setEncoding('utf8');
        res.on('data', chunk => {
            body.push(chunk);
        });
        res.on('end', () => {
            console.log('results', body.join(''));
            callback(null, body.join(''));
        });
    });

    req.on('error', err => {
        console.log('problem with request:', err.message);
    });

    const params = {
        Bucket,
        Key: event.Records[0].s3.object.key
    }
    S3.getObject(params, function(err, data){
        const payload = {
            "requests": [{
                "image": {
                    "content": data.Body.toString('base64')
                },
                "features": [{
                    "type": "LABEL_DETECTION",
                    "maxResults": 10
                },{
                    "type": "TEXT_DETECTION",
                    "maxResults": 10
                }]
            }]
        };

        req.write(JSON.stringify(payload));
        req.end();
    });
}