[英]mrjob virtualenv error in Hadoop cluster: Permission denied
我在一個擁有Hadoop集群的大型企業組織中工作。 我得到了管理員在所有Hadoop工作節點上安裝virtualenv
mrjob
以便我可以提交具有工作節點上可能不存在的標准Python
依賴關系的mrjob
。 根據此處的文檔,這就是我的mrjob.conf
文件的樣子:
runners:
hadoop:
setup:
- virtualenv venv
- . venv/bin/activate
- pip install nltk
我有一個使用nltk
包的簡單工作。 我可以驗證此安裝腳本是否在工作程序節點上運行(我可以將簡單的命令(例如,將一些數據寫入/tmp
的文件中,然后可以正常工作))。 但是,出現以下錯誤:
New python executable in venv/bin/python
Installing setuptools............done.
Installing pip...
Error [Errno 13] Permission denied while executing command /storage5/hadoop/map...env/bin/easy_install /usr/share/python-virtualenv/pip-1.1.tar.gz
...Installing pip...done.
Traceback (most recent call last):
File "/usr/bin/virtualenv", line 3, in <module>
virtualenv.main()
File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 938, in main
never_download=options.never_download)
File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 1054, in create_environment
install_pip(py_executable, search_dirs=search_dirs, never_download=never_download)
File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 643, in install_pip
filter_stdout=_filter_setup)
File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 976, in call_subprocess
cwd=cwd, env=env)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:572)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
是什么導致此錯誤?
感謝您將包部署到集群的想法。
至於您的問題,我認為它似乎沒有寫目錄的權限。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.