![](/img/trans.png)
[英]How to debug external .py functions run from Jupyter/IPython notebook
[英]Run MRJob from IPython notebook
我试图从IPython笔记本运行mrjob示例
from mrjob.job import MRJob
class MRWordFrequencyCount(MRJob):
def mapper(self, _, line):
yield "chars", len(line)
yield "words", len(line.split())
yield "lines", 1
def reducer(self, key, values):
yield key, sum(values)
然后用代码运行它
mr_job = MRWordFrequencyCount(args=["testfile.txt"])
with mr_job.make_runner() as runner:
runner.run()
for line in runner.stream_output():
key, value = mr_job.parse_output_line(line)
print key, value
并得到错误:
TypeError: <module '__main__' (built-in)> is a built-in class
有没有办法从IPython笔记本运行mrjob?
我还没有找到“完美的方式”,但你可以做的一件事是创建一个笔记本单元格,使用%%file
魔术,将单元格内容写入文件:
%%file wordcount.py
from mrjob.job import MRJob
class MRWordFrequencyCount(MRJob):
def mapper(self, _, line):
yield "chars", len(line)
yield "words", len(line.split())
yield "lines", 1
def reducer(self, key, values):
yield key, sum(values)
然后让mrjob
在稍后的单元格中运行该文件:
import wordcount
reload(wordcount)
mr_job = wordcount.MRWordFrequencyCount(args=['example.txt'])
with mr_job.make_runner() as runner:
runner.run()
for line in runner.stream_output():
key, value = mr_job.parse_output_line(line)
print key, value
请注意,我调用了我的文件wordcount.py
并从wordcount
模块导入了类MRWordFrequencyCount
- 文件名和模块必须匹配。 Python也会缓存导入的模块,当你更改wordcount.py
文件时,iPython不会重新加载模块,而是使用旧的缓存模块。 这就是我把reload()
调用放在那里的原因。
参考: https : //groups.google.com/d/msg/mrjob/CfdAgcEaC-I/8XfJPXCjTvQJ
更新(更短)
对于较短的第二个笔记本单元,您可以通过从笔记本中调用shell来运行mrjob
! python mrjob.py shakespeare.txt
参考: http ://jupyter.cs.brynmawr.edu/hub/dblank/public/Jupyter%20Magics.ipynb
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.