In the Reducer
of my MapReduce program, I wish to read a JobCounter
and a FileSystemCounter
. When running the command mapred job -status <job id>
, the counters I need are listed by their display names:
...
File System Counters
FILE: Number of bytes read=148874
FILE: Number of bytes written=22010065
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=135823
HDFS: Number of bytes written=44423504133
HDFS: Number of read operations=2185
HDFS: Number of large read operations=0
HDFS: Number of write operations=1316
Job Counters
Launched map tasks=1
Launched reduce tasks=200
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5293
Total time spent by all reduces in occupied slots (ms)=972893
Total time spent by all map tasks (ms)=5293
Total time spent by all reduce tasks (ms)=972893
Total vcore-seconds taken by all map tasks=5293
Total vcore-seconds taken by all reduce tasks=972893
Total megabyte-seconds taken by all map tasks=5420032
Total megabyte-seconds taken by all reduce tasks=996242432
...
How can I access these counters at runtime from within my Reducer
's code?
Using Google, I fail to find any useful information on how to access these counters. The straightforward attempt using Context.getCounter(String groupName, String counterName) fails to retrieve a Counter
instance and therefore throws a NullPointerException
upon calling getValue()
:
long bytes = context.getCounter(
FileSystemCounter.class.getName(),
FileSystemCounter.BYTES_WRITTEN.name()
).getValue();
long milliseconds = context.getCounter(
JobCounter.class.getName(),
JobCounter.MILLIS_REDUCES.name()
).getValue();
Counters counters = job.getCounters();
for (CounterGroup group : counters) {
System.out.println("* Counter Group: " + group.getDisplayName() + " (" + group.getName() + ")");
System.out.println(" number of counters in this group: " + group.size());
for (Counter counter : group) {
System.out.println(" - " + counter.getDisplayName() + ": " + counter.getName() + ": "+counter.getValue());
}
}
I think this will help to print all counters and their values.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.