I ran into a problem running the program on a cluster and decided to read from hdfs file in functions map and reduce. How to read line by line hdfs file and burn to read rows in ArrayList?
Just a code snippet for demonstration:
Path path = new Path(filePath);
FileSystem fs = path.getFileSystem(context.getConfiguration()); // context of mapper or reducer
FSDataInputStream fdsis = fs.open(path);
BufferedReader br = new BufferedReader(new InputStreamReader(fdsis));
String line = "";
ArrayList<String> lines = new ArrayList<String>();
while ((line = br.readLine()) != null) {
lines.add(line);
}
br.close();
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.