简体   繁体   中英

Node.js worker threads stop parallelizing when I require a Typescript-compiled file

So my long-term goal here is to run a function from "main.ts" in parallel with 7 different inputs. There's no shared resources, just a pure function.

As a test, I spun up worker threads that did trivial math operations over and over, and it parallelized perfectly (code shown below).

However, when I so much as require (not run in any way) a function from the main portion of my app, the trivial math operations stop parallelizing.

I'm completely lost on how requiring a file can change thread behavior. Anyone have ideas? I've pasted as much info as I can below.

worker_thread.js

const process = require("process");

function heavyComputation(data){
  console.time(data.toString())
  let sum = 0;
  for (let i = 0; i < 100000000; i++){
    sum = Math.pow(sum,1.02) % 100000;
  }
  console.timeEnd(data.toString())
  return data + 1000;
}

process.on("message", (message) => {
  process.send({
    result: heavyComputation(1),
  });
});

worker_test.js

const child_process = require("child_process");

let workers = [];
const NUM_THREADS = 7;
let pendingResults = NUM_THREADS;

console.time("async");
function onMessage(message) {
  // console.log("Received response message:", message.result);
  pendingResults--;
  if (pendingResults == 0){
    console.timeEnd("async");
    workers.forEach(x => x.kill());
  }
}

for (let i = 0; i < NUM_THREADS; i++){
  workers[i] = child_process.fork("src/server/worker_thread.js");
  workers[i].addListener("message", onMessage);
}

for (let i = 0; i < NUM_THREADS; i++){
  workers[i].send({ data: argsData });
}

The above two files work exactly as you'd expect:

before

1: 843.353ms
1: 837.07ms
1: 848.494ms
1: 844.644ms
1: 847.34ms
1: 855.917ms
1: 896.467ms
async: 976.024ms

However, when I add the following to worker_thread.js , it all breaks. What it's importing is the transpiled version of my main typescript file.

const mainApp = require("../../built/src/server/main.js");

after

1: 846.826ms
1: 864.081ms
1: 873.927ms
1: 874.493ms
1: 921.775ms
1: 927.178ms
1: 942.86ms
async: 2.205s

Notice how the whole operation takes significantly longer, despite each individual one taking the same amount of time as before.

The contents of main are far too large to post here, but here's some of the imports/config at the top of the built file:

"use strict";
var __assign = (this && this.__assign) || function () {
...

Object.defineProperty(exports, "__esModule", { value: true });
exports.addTapInfoToAiParams = exports.evaluateFirstPlacements = exports.getBestMove = void 0;
var evaluator = require("./evaluator");
var aiModeManager = require("./ai_mode_manager");
var boardHelper = require("./board_helper");

So it turns out that deep in the dependency tree of the file I was requiring, it was reading a large text file from hard disk, which slowed down the worker thread significantly, and made multithreading harder.

Mystery solved. I doubt many other people will have this issue, but in case someone does, hopefully this saved you some time.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM