如何将hdfs文件读入apache梁？

Question

I'm not able to understand how to read it with TextIO. Can anyone please explain it with examples?

Answer 1

Try the following. It worked in my case.

String[] args1 = new String[] {
    "--hdfsConfiguration=[{\"fs.defaultFS\":\"hdfs://localhost:9000\"}]" };

HadoopFileSystemOptions options =
    PipelineOptionsFactory.fromArgs(args1).withValidation().as(
        HadoopFileSystemOptions.class);

Pipeline pipeline = Pipeline.create(options);
pipeline.apply(
    TextIO.read().from("hdfs://localhost:9000/path-to-file/*"));