使用Node.JS spawn生成多个文件的bash cli

Question

I am creating a program in Node.JS that extract pdf text using the command-line utility pdftotext by creating child_process.spawn for each file. I would like to know if this process is CPU heavy and if it is possible thousands of people to use without breaks anything.

Is create a child_process is heavy? If pdftotext is not multithreading, how can I scale? Do i need load balancing?

Thanks.

Answer 1

Let's break this down a bit:

I would like to know if this process is CPU heavy

I am not sure how CPU intense pdftotext is for a single file. That would also depend on how big each file is, but generally speaking and since the action of extracting PDF to text has no asynchronous work and is CPU bound, I would imagine the process to be CPU heavy, specially with lots of load.

and if it is possible thousands of people to use without breaks anything.

Spawning a new process for every single file or on every single request is generally not a good idea. Spawning a process is an expensive operation that requires a lot of memory. Having thousands of people using your service at the same time would require thousands of processes to be open simultaneously on your server which would cause memory to choke and your server would max at a certain limit and fail after that.

Is create a child_process is heavy? If pdftotext is not multithreading, how can I scale? Do i need load balancing?

As mentioned, spawning a new process is never a cheap operation. It requires memory and resources.

Every file will run in a separate process. Weather pdftotext is implemented to open a single or multiple threads in a process is irrelevant here, either way the process with all it's threads will be competing for machine resources with other processes. Of course it is beneficial if it is implemented in a way that divides work among different threads and can execute in parallel as this makes it faster, however what you would be more concerned about is how long it takes to extract text from a single file ie how long the process spends executing.

If you are to run this as a service, you would need to benchmark, optimize and for sure depending on the load you want to support and benchmark results, have to load balance between a few high end machines.

I hope I managed to answer some of your questions.