[英]How do I identify that my Foundry job's stage has skew?
I have a job running with a stage that seems to be taking a long time.我有一份工作在一个似乎需要很长时间的舞台上运行。 I've heard that this might be due to something called 'skew'.
我听说这可能是由于一种叫做“偏斜”的东西。
How do I know if I'm being impacted by this?我怎么知道我是否受到此影响?
I know this is commonly associated with joins, windows, and other operations that incur shuffles but I don't know how to identify it.我知道这通常与连接、windows 和其他导致随机播放的操作相关联,但我不知道如何识别它。
In the above example, there is a task in this job + stage that is taking orders of magnitude longer to run because its input size is orders of magnitude larger than the other tasks.在上面的示例中,此作业 + 阶段中有一个任务的运行时间要长几个数量级,因为它的输入大小比其他任务大几个数量级。
This is the definition of a skewed task / skewed stage.这是倾斜任务/倾斜阶段的定义。
If you want to know what value is causing this task to be slow, check out the guidance over here如果您想知道什么值导致此任务变慢,请查看此处的指南
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.