How can I limit the number of child processes in imagemin-mozjpeg?

Question

I'm using imagemin-mozjpeg which uses mozjpeg binary to compress images.

The problem is that I'm using it in a nodejs webserver.

This is how it works now:

1. I'm uploading a JPEG image using "request" module (fs.createReadStream).

2. Multer process the stream the stream and keep it in buffer (memory storage).

3. Then pass the buffer to the imagemin to compress.

4. The compressed buffer is then written to a file. (example.jpg)

Everything works.

The problem here is that for each request, a new child process of mozjpeg binary is spawned, cjpeg.

1 child process is consuming 12.5 MB of memory (for a .5 MB file).

If I've 50 requests at the same time, that goes to ~700 MB because for 50 images there are 50 child processes.

Is there a way that I can limit the the number of child processes? (the library is using "execa" module) or just spawn 4-5 child process and they do the compression for all the requests.

Thanks

if (req.files.myimage[0].buffer) {
            let fileNumber = filesUploaded++;

            imagemin.buffer(req.files.myimage[0].buffer, {
                plugins: [
                    imageminMozjpeg({ quality: 60, progressive: true })
                ]
            })
                .then(file => {
                    fs.writeFile(__dirname + '/uploads/' + uuidv4() + '.jpg', file, 'hex' , function () {
                        res.end("SUCCESS" + fileNumber.toString());
                    });

                })
                .catch(err => {
                    console.log(err);
                    res.end("FAILED");
                });

        }

Answer 1

The main concept to solve this problem is to limit the number of call to the imagemin() (who spawns image processing process).

Maybe you can implement a task scheduling system using a task queue to gather requests and some workers to process the request with imagemin() .

    var multer  = require('multer')
    var upload = multer({ dest: 'your/uploads/' })

    // TaskScheduler, a wrapper of a task array
    class TaskScheduler extends Array {
      constructor (MAX_SLOTS, worker) {
        super()
        this._MAX_SLOTS= MAX_SLOTS
        this._busy_slots= 0
        this._worker= worker
      }

      /**
       * Create new tasks
       */
      push (...tasks) {
        const ret = super.push(...tasks)

        this.run()
        return ret
      }

      /**
       * Run tasks in available slots
       */
      run () {
        // if there are any tasks and available slots
        while (this.length > 0 && this._busy_slots < this._MAX_SLOTS) {
          const firstTask = this.shift()
          this._worker(firstTask).then(() => {
            // release the slot
            this._busy_slots--

            // since a task slot is released
            // call run() again to process another task from the queue
            this.run()
          })
          this._busy_slots++
        }
      }
    }

    // worker is supposed to return a Promise
    const scheduler = new TaskScheduler(5, function (task) {
      return imagemin.buffer(task.buffer, { /* imagemin options */ })
        .then(() => { /* write image files */ })
        .catch(() => { /* error handling */ })
    })

    // schedule the task when there is an incoming request
    // the actual code depends on your web server
    // here what to do in the callback is the point ;)

    // Take express for example, `app` is the express.Application
    app.post('your/end/point', upload.fields([{ name: 'myimage' }]), function (req) {
      if (req.files.myimage[0]) {
        scheduler.push(req.files.myimage[0])
      }
    })

Note that since your scheduler is extended from Array , you can use any Array method to manage your tasks, eg pop() to discard the last task, shift() to discard the first task and unshift(newTask) to insert a new task at the begining of the scheduler queue.