[英]How does Apache-Spark work with methods inside a class
我現在正在學習Apache-Spark。 在仔細閱讀了Spark教程之后,我了解了如何將Python函數傳遞給Apache-Spark來處理RDD數據集。 但是現在我仍然不知道Apache-Spark如何與類中的方法一起使用。 例如,我的代碼如下:
import numpy as np
import copy
from pyspark import SparkConf, SparkContext
class A():
def __init__(self, n):
self.num = n
class B(A):
### Copy the item of class A to B.
def __init__(self, A):
self.num = copy.deepcopy(A.num)
### Print out the item of B
def display(self, s):
print s.num
return s
def main():
### Locally run an application "test" using Spark.
conf = SparkConf().setAppName("test").setMaster("local[2]")
### Setup the Spark configuration.
sc = SparkContext(conf = conf)
### "data" is a list to store a list of instances of class A.
data = []
for i in np.arange(5):
x = A(i)
data.append(x)
### "lines" separate "data" in Spark.
lines = sc.parallelize(data)
### Parallelly creates a list of instances of class B using
### Spark "map".
temp = lines.map(B)
### Now I got the error when it runs the following code:
### NameError: global name 'display' is not defined.
temp1 = temp.map(display)
if __name__ == "__main__":
main()
實際上,我使用上述代碼使用temp = lines.map(B)
並行生成class B
的實例列表。 之后,我做了temp1 = temp.map(display)
,因為我想並行打印出class B
實例列表中的每個項目。 但是現在出現了錯誤: NameError: global name 'display' is not defined.
我想知道如果仍然使用Apache-Spark並行計算,如何解決該錯誤。 如果有人幫助我,我真的很感激。
結構體
.
├── ab.py
└── main.py
import numpy as np
from pyspark import SparkConf, SparkContext
import os
from ab import A, B
def main():
### Locally run an application "test" using Spark.
conf = SparkConf().setAppName("test").setMaster("local[2]")
### Setup the Spark configuration.
sc = SparkContext(
conf = conf, pyFiles=[
os.path.join(os.path.abspath(os.path.dirname(__file__)), 'ab.py')]
)
data = []
for i in np.arange(5):
x = A(i)
data.append(x)
lines = sc.parallelize(data)
temp = lines.map(B)
temp.foreach(lambda x: x.display())
if __name__ == "__main__":
main()
import copy
class A():
def __init__(self, n):
self.num = n
class B(A):
### Copy the item of class A to B.
def __init__(self, A):
self.num = copy.deepcopy(A.num)
### Print out the item of B
def display(self):
print self.num
評論:
for x in rdd.sample(False, 0.001).collect(): x.display()
foreach
而不是map
display
方法。 我不知道什么應該是s
在這方面
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.