简体   繁体   English

在CDH4集群(Hadoop 2 + MapReduce v1)上运行Pig时无法进入mapreduce模式

[英]Cannot get into mapreduce mode while running Pig on CDH4 cluster (Hadoop 2 + MapReduce v1)

I want to run pig scripts (non-embedded and embedded) on a CDH4 cluster of 3 Amazon instances. 我想在3个Amazon实例的CDH4集群上运行Pig脚本(非嵌入式和嵌入式)。 I created a fake configuration file (but the addresses are correct) for pig located at /home/ubuntu/core-site.xml that looks like this: 我为位于/home/ubuntu/core-site.xml的Pig创建了一个伪造的配置文件(但地址正确),如下所示:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://server1:8020</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>server1:8021</value>
  </property>
</configuration>

When I tried to run: 当我尝试运行时:

ubuntu@server1:~$ export HADOOP_CONF_DIR=/home/ubuntu
ubuntu@server1:~$ export PIG_CLASSPATH=/home/ubuntu
ubuntu@server1:~$ pig -x mapreduce -f test.pig 

The script ran and returned correct result but in the console I found a lot of "LocalJobRunner" and in MapReduce job tracker web interface no job is reported. 该脚本已运行并返回了正确的结果,但是在控制台中我发现了很多“ LocalJobRunner”,并且在MapReduce作业跟踪器Web界面中未报告任何作业。 Can any one tell me why doesn't it run in mapreduce mode and why doesn't it report any error of that situation? 谁能告诉我为什么它不以mapreduce模式运行,为什么它不报告这种情况的任何错误? How can I run it in mapreduce mode? 如何在mapreduce模式下运行它?

My cluster is 4.1.3 (freshly installed, all configuration is default) with Pig 0.10.0-cdh4.1.3. 我的集群是Pig 1.30.0-cdh4.1.3(全新安装,默认为所有配置)。

2013-02-20 18:57:15,843 [main] INFO  org.apache.pig.Main - Apache Pig version 0.10.0-cdh4.1.3 (rexported) compiled Jan 26 2013, 17:35:45
2013-02-20 18:57:15,843 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/ubuntu/epi-tre/pig_1361386635821.log
2013-02-20 18:57:16,519 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://server1:8020
2013-02-20 18:57:16,524 [main] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:17,249 [main] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:17,826 [main] WARN  org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.permissions.supergroup is deprecated. Instead, use dfs.permissions.superusergroup
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.interval is deprecated. Instead, use dfs.namenode.replication.interval
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.access.time.precision is deprecated. Instead, use dfs.namenode.accesstime.precision
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.http.address is deprecated. Instead, use dfs.namenode.http-address
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.considerLoad is deprecated. Instead, use dfs.namenode.replication.considerLoad
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.write.packet.size is deprecated. Instead, use dfs.client-write-packet-size
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.permissions is deprecated. Instead, use dfs.permissions.enabled
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.block.size is deprecated. Instead, use dfs.blocksize
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.name.dir.restore is deprecated. Instead, use dfs.namenode.name.dir.restore
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.https.need.client.auth is deprecated. Instead, use dfs.client.https.need-auth
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.backup.http.address is deprecated. Instead, use dfs.namenode.backup.http-address
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.secondary.http.address is deprecated. Instead, use dfs.namenode.secondary.http-address
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.safemode.extension is deprecated. Instead, use dfs.namenode.safemode.extension
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.df.interval is deprecated. Instead, use fs.df.interval
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.https.client.keystore.resource is deprecated. Instead, use dfs.client.https.keystore.resource
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.datanode.max.xcievers is deprecated. Instead, use dfs.datanode.max.transfer.threads
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - topology.script.number.args is deprecated. Instead, use net.topology.script.number.args
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.balance.bandwidthPerSec is deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.safemode.threshold.pct is deprecated. Instead, use dfs.namenode.safemode.threshold-pct
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.period is deprecated. Instead, use dfs.namenode.checkpoint.period
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2013-02-20 18:57:18,102 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY
2013-02-20 18:57:18,460 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-02-20 18:57:18,475 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2013-02-20 18:57:18,507 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2013-02-20 18:57:18,508 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2013-02-20 18:57:18,530 [main] WARN  org.apache.hadoop.conf.Configuration - session.id is deprecated. Instead, use dfs.metrics.session-id
2013-02-20 18:57:18,531 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
2013-02-20 18:57:18,553 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-02-20 18:57:18,563 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-02-20 18:57:18,566 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2458140745309154844.jar
2013-02-20 18:57:22,423 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2458140745309154844.jar created
2013-02-20 18:57:22,453 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-02-20 18:57:22,544 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-02-20 18:57:22,553 [Thread-4] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2013-02-20 18:57:22,581 [Thread-4] WARN  org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2013-02-20 18:57:22,735 [Thread-4] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:22,735 [Thread-4] WARN  org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:22,790 [Thread-4] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-02-20 18:57:22,790 [Thread-4] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2013-02-20 18:57:22,851 [Thread-4] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2013-02-20 18:57:23,045 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-02-20 18:57:23,055 [Thread-4] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:23,057 [Thread-4] WARN  org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:23,141 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter set in config null
2013-02-20 18:57:23,178 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.df.interval is deprecated. Instead, use fs.df.interval
2013-02-20 18:57:23,178 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects
2013-02-20 18:57:23,187 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2013-02-20 18:57:23,187 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.block.size is deprecated. Instead, use dfs.blocksize
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.access.time.precision is deprecated. Instead, use dfs.namenode.accesstime.precision
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.considerLoad is deprecated. Instead, use dfs.namenode.replication.considerLoad
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.balance.bandwidthPerSec is deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.safemode.threshold.pct is deprecated. Instead, use dfs.namenode.safemode.threshold-pct
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.http.address is deprecated. Instead, use dfs.namenode.http-address
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.name.dir.restore is deprecated. Instead, use dfs.namenode.name.dir.restore
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.https.client.keystore.resource is deprecated. Instead, use dfs.client.https.keystore.resource
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.backup.http.address is deprecated. Instead, use dfs.namenode.backup.http-address
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.permissions is deprecated. Instead, use dfs.permissions.enabled
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.safemode.extension is deprecated. Instead, use dfs.namenode.safemode.extension
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.datanode.max.xcievers is deprecated. Instead, use dfs.datanode.max.transfer.threads
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.https.need.client.auth is deprecated. Instead, use dfs.client.https.need-auth
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.interval is deprecated. Instead, use dfs.namenode.replication.interval
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.write.packet.size is deprecated. Instead, use dfs.client-write-packet-size
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.permissions.supergroup is deprecated. Instead, use dfs.permissions.superusergroup
2013-02-20 18:57:23,191 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - topology.script.number.args is deprecated. Instead, use net.topology.script.number.args
2013-02-20 18:57:23,191 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.secondary.http.address is deprecated. Instead, use dfs.namenode.secondary.http-address
2013-02-20 18:57:23,191 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.period is deprecated. Instead, use dfs.namenode.checkpoint.period
2013-02-20 18:57:23,191 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl
2013-02-20 18:57:23,191 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:23,193 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter is org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter
2013-02-20 18:57:23,246 [Thread-5] WARN  mapreduce.Counters - Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-02-20 18:57:23,333 [Thread-5] INFO  org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
2013-02-20 18:57:23,344 [Thread-5] INFO  org.apache.hadoop.mapred.Task -  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3bfc47
2013-02-20 18:57:23,362 [Thread-5] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed hdfs://server1:8020/tmp/cf_rating.10000.csv:0+676168
2013-02-20 18:57:23,374 [Thread-5] INFO  org.apache.hadoop.mapred.MapTask - io.sort.mb = 100
2013-02-20 18:57:23,507 [Thread-5] INFO  org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720
2013-02-20 18:57:23,507 [Thread-5] INFO  org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680
2013-02-20 18:57:23,639 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
2013-02-20 18:57:24,485 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
2013-02-20 18:57:24,488 [Thread-5] INFO  org.apache.hadoop.mapred.MapTask - Starting flush of map output
2013-02-20 18:57:25,059 [Thread-5] INFO  org.apache.hadoop.mapred.MapTask - Finished spill 0
2013-02-20 18:57:25,065 [Thread-5] INFO  org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
2013-02-20 18:57:25,087 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
2013-02-20 18:57:25,098 [Thread-5] INFO  org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
2013-02-20 18:57:25,104 [Thread-5] WARN  mapreduce.Counters - Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-02-20 18:57:25,130 [Thread-5] INFO  org.apache.hadoop.mapred.Task -  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@461d318f
2013-02-20 18:57:25,130 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
2013-02-20 18:57:25,137 [Thread-5] INFO  org.apache.hadoop.mapred.Merger - Merging 1 sorted segments
2013-02-20 18:57:25,164 [Thread-5] INFO  org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 25 bytes
2013-02-20 18:57:25,164 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
2013-02-20 18:57:25,336 [Thread-5] INFO  org.apache.hadoop.mapred.Task - Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
2013-02-20 18:57:25,337 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:25,338 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:25,339 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
2013-02-20 18:57:25,341 [Thread-5] INFO  org.apache.hadoop.mapred.Task - Task attempt_local_0001_r_000000_0 is allowed to commit now
2013-02-20 18:57:25,371 [Thread-5] INFO  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://server1:8020/tmp/temp-1313079537/tmp188373172
2013-02-20 18:57:25,371 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
2013-02-20 18:57:25,373 [Thread-5] INFO  org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_r_000000_0' done.
2013-02-20 18:57:28,144 [main] WARN  org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for job job_local_0001
2013-02-20 18:57:28,147 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-02-20 18:57:28,151 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 

HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
2.0.0-cdh4.1.3  0.10.0-cdh4.1.3 ubuntu  2013-02-20 18:57:18 2013-02-20 18:57:28 GROUP_BY

Success!

Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime  MinMapTIme  AvgMapTime  MaxReduceTime   MinReduceTimeAvgReduceTime  Alias   Feature Outputs
job_local_0001  1   1   n/a n/a n/a n/a n/a n/a a,b,c   GROUP_BY,COMBINER   hdfs://server1:8020/tmp/temp-1313079537/tmp188373172,

Input(s):
Successfully read 0 records from: "/tmp/cf_rating.10000.csv"

Output(s):
Successfully stored 0 records in: "hdfs://server1:8020/tmp/temp-1313079537/tmp188373172"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_local_0001


2013-02-20 18:57:28,151 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2013-02-20 18:57:28,158 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-02-20 18:57:28,159 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(10000)

原来是我忘了部署MapReduce“客户端配置”,所以我的整个集群都以本地模式运行:“>已解决。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM