如何使用 Bull.js 队列返回生成的图像？

Question

My use case is this: I want to create screenshots of parts of a page. For technical reasons, it cannot be done on the client-side (see related question below) but needs puppeteer on the server.

As I'm running this on Heroku, I have the additional restriction of a quite small timeout window. Heroku recommends therefore to implement a queueing system based on bull.js and use worker processes for longer-running tasks as explained here .

I have two endpoints (implemented with Express), one that receives a POST request with some configuration JSON, and another one that responds to GET when provided with a job identifier (slightly modified for brevity):

This adds the job to the queue:

router.post('/', async function(req, res, next) {
    let job = await workQueue.add(req.body.chartConfig)
    res.json({ id: job.id })
})

This returns info about the job

router.get('/:id', async(req, res) => {
    let id = req.params.id;
    let job = await workQueue.getJob(id);
    
    let state = await job.getState();
    let progress = job._progress;
    let reason = job.failedReason;
    res.json({ id, state, progress, reason });
})

In a different file:

const start = () => {
    let workQueue = new queue('work', REDIS_URL);
    workQueue.process(maxJobsPerWorker, getPNG)
}

const getPNG = async(job) => {

    const { url, width, height, chart: chartConfig, chartUrl } = job.data

    // ... snipped for brevity

    const png = await page.screenshot({
        type: 'png',
        fullPage: true
    })
    await page.close()
    job.progress(100)
    return Promise.resolve({ png })
}

// ...

throng({ count: workers, worker: start })

module.exports.getPNG = getPNG

The throng invocation at the end specifies the start function as the worker function to be called when picking a job from the queue. start itself specifies getPNG to be called when treating a job.

My question now is: how do I get the generated image ( png )? I guess ideally I'd like to be able to call the GET endpoint above which would return the image, but I don't know how to pass the image object.

As a more complex fall-back solution I could imagine posting the image to an image hosting service like imgur, and then returning the URL upon request of the GET endpoint. But I'd prefer, if possible, to keep things simple.

This question is a follow-up from this one:

Issue with browser-side conversion SVG -> PNG

Answer 1

I've opened a ticket on the GitHub repository of the bull project. The developers said that the preferred practice is to store the binary object somewhere else, and to add only the link metadata to the job's data store.

However, they also said that the storage limit of a job object appears to be 512 Mb. So it is also quite possible to store an image of a reasonable size as a base64-encoded string.