如何使用Ruby在Hadoop HDFS中编写和读取文件？

Question

Is there a way to work with HDFS Api using Ruby? As I can understand there is no multilanguage file Api and the only way is to use native Java Api. I tried using JRuby but this solution is to unstable and not very native. Also I looked at HDFS Thrift Api but it's not complete and also lacks many features (like writing to indexed files).

Is there a way to work with HDFS using Ruby besides from using JRuby or Thrift Api?

Answer 1

There are two projects in github that fit what you're asking. ruby-hdfs provides native C bindings to HDFS for Ruby. ganapati interfaces with a Thrift server.

You could also make system calls directly to the file system shell . For example:

cmd = "hadoop fs -mkdir #{hdfs_path}"
cmd += " 2> /dev/null"
system(cmd)
if $? == 0
  puts 'ok'
  exit(0)
else
  puts "Error: failed to create hdfs://#{hdfs_path}"
  exit(2)
end