简体繁体 English

Dask.distributed 杀死工人每次我跑步

[英]Dask.distributed kill workers Everytime I run

原文 2020-07-22 22:54:42 1 1 python/ dask/ dask-distributed

I have been having problems with dask.distributed.我一直遇到 dask.distributed 的问题。 Everytime I run it, it always has memory issues and kills my workers, when running on my laptop.每次我运行它时，它总是有 memory 问题并杀死我的工人，在我的笔记本电脑上运行时。 I have a laptop with 16GB and run 4 workers, when I try to run a computation it seems to straight away kill workers.我有一台 16GB 的笔记本电脑并运行 4 个工作人员，当我尝试运行计算时，它似乎立即杀死了工作人员。 Ive tried limiting the ram to 3.5 GB but that doesnt work.我尝试将内存限制为 3.5 GB，但这不起作用。 My chunk sizes are around 200MB.我的块大小约为 200MB。 I am using a zarr file with a 3diim array stored.我正在使用存储了 3diim 数组的 zarr 文件。 So I don't understand why I have this issue.所以我不明白为什么我有这个问题。 I had it working a few months ago and took some time away from it when this problem occurred.几个月前我让它工作了，当这个问题发生时，我花了一些时间离开它。 I tired a smaller overall data set and the same issue occurs.我厌倦了一个较小的整体数据集，并且发生了同样的问题。 Any thoughts?有什么想法吗？ When I run this outside of distributed I don't have the same issue?当我在分布式之外运行它时，我没有同样的问题吗？

Happy to show error reports and a minimum example if anyone has idea's?如果有人有想法，很高兴显示错误报告和最小示例？

1 个解决方案

I recommend looking at logs.我建议查看日志。 KilledWorker errors can occur for many reasons.发生KilledWorker错误的原因有很多。 For example you might not have the same version of Dask everywhere or you might be launching workers someplace where they can't connect.例如，您可能没有在所有地方都使用相同版本的 Dask，或者您可能在无法连接的地方启动工作人员。