Hadoop - streaming data from HTTP upload (PUT) into HDFS directly

Question

I have the following application deployment

• web front-end taking data from client through HTTP/FTP
• hadoop cluster

I need to store client's data on HDFS. What is the best way of doing that? Is it possible to stream data to HDFS directly, without consuming all data from the client on local drive, and then put it into the HDFS?

Answer 1

The feasible options which I can think of right now are :

• HttpFS
• WebHDFS
• FTP client over HDFS
• HDFS over WebDAV

Choosing the "best" one is totally upto you, based on your convenience and ease.

Answer 2

Personally, if you want low latency access to HDFS, your best bet is HBase. You can put and get values very easily since it is just a key value store. We are using the same thing in our application(s) and it works fabulously.