Spark streaming and mocking hdfs

Question

There is a requirement to implement a test for a spark streaming code. This particular code is running in a separate jvm by using this library And the input for above application is hdfs. I've started MiniDFSCluster like in this example (java version) But i don't think it will work because these are in two different JVMs.

What would be the best approach to mock the hdfs input if i were to successfully test the spark streaming code.

I explained above scenario generally. The real requirement is to implement a successful cucumber test.

Answer 1

可以尝试在本地模式下运行Spark并指定诸如“ file：/// foo / bar”之类的路径，而不是尝试模拟hdfs-然后将使用本地文件系统代替hdfs。