[英]How to save file in hadoop with python
I am trying to save file in Hadoop with python 2.7. 我正在尝试使用python 2.7将文件保存在Hadoop中。 I searched on the internet.
我在互联网上搜索。 I got some code to save a file in Hadoop but it is taking the entire folder while saving (total files in the folder are saving in Hadoop).
我有一些代码可以将文件保存在Hadoop中,但是在保存时会占用整个文件夹(文件夹中的所有文件都保存在Hadoop中)。 But I need to save a specific file.
但是我需要保存一个特定的文件。
Here is the link to save a folder in Hadoop: http://www.hadoopy.com/en/latest/tutorial.html#putting-data-on-hdfs 以下是在Hadoop中保存文件夹的链接: http : //www.hadoopy.com/en/latest/tutorial.html#putting-data-on-hdfs
Now what I need is save a particular file in Hadoop like abc.txt
. 现在,我需要在Hadoop中保存一个特定文件,例如
abc.txt
。
Here is my code: 这是我的代码:
import hadoopy
hdfs_path = 'hdfs://192.168.x.xxx:xxxx/video/py5'
def main():
local_path = open('abc.txt').read()
hadoopy.writetb(hdfs_path, local_path)
if __name__ == '__main__':
main()
Here i am getting need more than one value to unpack
在这里,我
need more than one value to unpack
Any help would be appreciated. 任何帮助,将不胜感激。
The hadoopy.writetb
seems to expects an iterable of two-values as its second argument. hadoopy.writetb
似乎希望将两个值的可迭代值作为其第二个参数。 Try: 尝试:
hadoopy.writetb(hdfs_path, [("abc.txt", open("abc.txt").read())])
http://www.hadoopy.com/en/latest/api.html?highlight=hadoopy.writetb#hadoopy.writetb http://www.hadoopy.com/en/latest/api.html?highlight=hadoopy.writetb#hadoopy.writetb
writedb requires second arg as kvs – Iterator of (key, value) writedb需要第二个arg作为kvs –(键,值)的迭代器
As per the link you have given, you have forgot to copy the function read_local_dir
in your code. 按照您给出的链接,您已经忘记在代码中复制函数
read_local_dir
。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.