Spark Streaming检查点到远程HDFS

Question

I am trying to checkpoint my spark streaming context to hdfs to handle a failure at some point of my application. I have my HDFS setup on a separate cluster and spark running on a separate standalone server. To do this, I am using :

ssc.checkpoint(directory: String)

This gives me : org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE when I try with directory as "hdfs://hostname:port/pathToFolder"

How can I checkpoint to a remote hdfs path? Is it possible to add credentials to the string uri? I tried googling, but no help so far.

Thanks and appreciate any help!

Answer 1

您可以使用以下方式提供凭据：

hdfs://username:password@hostname:port/pathToFolder