如何使用Java有效地读取Hadoop（HDFS）文件中的第一行？

Question

I have a large CSV file on my Hadoop cluster. The first line of the file is a 'header' line, which consists of field names. I want to do an operation on this header line, but I do not want to process the whole file. Also, my program is written in Java and using Spark.

What is an efficient way to read just the first line of a large CSV file on an Hadoop cluster?

Answer 1

You can access hdfs files with FileSystem class and friends:

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;

DistributedFileSystem fileSystem = new DistributedFileSystem();
Configuration conf = new Configuration();
fileSystem.initialize(new URI("hdfs://namenode-host:54310"), conf);
FSDataInputStream input = fileSystem.open(new Path("/path/to/file.csv"));
System.out.println((new BufferedReader(new InputStreamReader(input))).readLine());

This code won't use MapReduce and will run with a reasonable speed.