如何使用本地输入文件在本地模式下运行烫伤测试

Question

Scalding has a great utility to run an integration test for the job flow. In this way the inputs and outputs are the in-memory buffer

val input = List("0" -> "This a a day")
val expectedOutput = List(("This", 1),("a", 2),("day", 1))
 JobTest(classOf[WordCountJob].getName)
  .arg("input", "input-data")
  .arg("output", "output-data")
  .source(TextLine("input-data"), input)
  .sink(Tsv("output-data")) {
  buffer: mutable.Buffer[(String, Int)] => {
    buffer should equal(expectedOutput)
  }
}.run

How can I transfare/write another code that will read input and write output to the real local file? Like FileTap/LFS in cascading - and not an in-memory approach

Answer 1

You might check out HadoopPlatformJobTest and the TypedParquetTupleTest.scala example which uses a local mini-cluster.

This unit test writes to a "MiniLocalCLuster" - While it's not directly a file, but accessible via reading the local minicluster with Hadoop filesystem.

Given you local file scenario, maybe you can copies the files with local reads to the mini-HDFS.