如何使用 Google Cloud Dataflow 增加 Apache Beam 管道工作线程上的线程堆栈大小？

Question

I'm getting a StackOverflowError on my Beam workers due to running out the thread stack, and because it's deep within the running of a SqlTransform it's not straightforward to reduce the number of calls being made.

Is it possible to change the JVM thread stack size for my workers, either through Google Cloud Dataflow or Beam's own pipeline options?

Answer 1

I don't think there's an easy way to do this.

• If this is an issue of stack trace being purged by Cloud Logging, may be it might be possible to catch the exception yourself and inspect that instead of just logging it.
• If this is an issue of default stack trace depth set by the JVM not being enough, I don't think there's a way to update this for Dataflow today unfortunately.