如何使用 Python pickle 将文件转储到 Hadoop HDFS 目录？

Question

I am on a VM in a directory that contains my Python (2.7) class. I am trying to pickle an instance of my class to a directory in my HDFS.

I'm trying to do something along the lines of:

import pickle

my_obj = MyClass() # the class instance that I want to pickle

with open('hdfs://domain.example.com/path/to/directory/') as hdfs_loc:
    pickle.dump(my_obj, hdfs_loc)

From what research I've done, I think something like snakebite might be able to help...but does anyone have more concrete suggestions?

Answer 1

If you use PySpark, then you can use the saveAsPickleFile method:

temp_rdd = sc.parallelize(my_obj)
temp_rdd.coalesce(1).saveAsPickleFile("/test/tmp/data/destination.pickle")

Answer 2

Here is a work around if you are running in a Jupyter notebook with sufficient permissions:

import pickle

my_obj = MyClass() # the class instance that I want to pickle
local_filename = "pickle.p"
hdfs_loc = "//domain.example.com/path/to/directory/"
with open(local_filename, 'wb') as f:
    pickle.dump(my_obj, f)
!!hdfs dfs -copyFromLocal $local_filename  $hdfs_loc

Answer 3

You can dump Pickle object to HDFS with PyArrow :

import pickle
import pyarrow as pa

my_obj = MyClass() # the class instance that I want to pickle

hdfs = pa.hdfs.connect()
with hdfs.open('hdfs://domain.example.com/path/to/directory/filename.pkl', 'wb') as hdfs_file:
    pickle.dump(my_obj, hdfs_file)