Spark RDD：在 range() object 处使用 collect()

Question

I want to convert the numbers 0 to 99 into a RDD.我想将数字 0 到 99 转换为 RDD。

rd1 = range(1, 100)
test = sc.parallelize(rd1)

When I use the collect() function...当我使用 collect() function ...

print(test.collect())

...I receive the following error message: ...我收到以下错误消息：

PicklingError: Could not pickle object as excessively deep recursion required.

According to this documentary, it's supposed to work.根据这部纪录片，它应该可以工作。 Can you tell me what I'm doing wrong?你能告诉我我做错了什么吗？

Thank you very much.非常感谢。

Answer 1

If someone else has the same problem.如果其他人有同样的问题。 I could solve it by selecting only the lines I want to execute.我可以通过只选择我想要执行的行来解决它。

I think that other scripts that ran parallel led to the error.我认为其他并行运行的脚本导致了错误。