简体   繁体   中英

Dataflow job on Java SDK 2.11.0 does not scale

I have a streaming job that reads from pubsub and writes to pubsub, doing some basic processing in the middle.
When working on SDK 2.5.0, and executing it "directly", it works as expected, and scale as needed from 1 worker to as many workers as it should. Trying to upgrade to latest version, 2.11.0, and trying to use templates - it works ok, but does not scale.
Attached are both jobs' pipeline options.
Any suggestion ? Anything that might have been changed between those 2 versions that I am missing ?
Thanks,
Shushu SDK 2.5.0示例 SDK 2.11.0示例

Clarification :
Even though I set 20 workers - I only get 1. 在此输入图像描述

numWorkers is set to 20. As such, when the job starts, 20 workers will be allocated based on this value.

maxNumWorkers is also set to 20. So, no more than 20 workers will ever be allocated.

If need the job to scale beyond the initial 20 workers, modify the maxNumWorkers value.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM