[英]Cannot get into mapreduce mode while running Pig on CDH4 cluster (Hadoop 2 + MapReduce v1)
I want to run pig scripts (non-embedded and embedded) on a CDH4 cluster of 3 Amazon instances. 我想在3个Amazon实例的CDH4集群上运行Pig脚本(非嵌入式和嵌入式)。 I created a fake configuration file (but the addresses are correct) for pig located at /home/ubuntu/core-site.xml that looks like this:
我为位于/home/ubuntu/core-site.xml的Pig创建了一个伪造的配置文件(但地址正确),如下所示:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://server1:8020</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>server1:8021</value>
</property>
</configuration>
When I tried to run: 当我尝试运行时:
ubuntu@server1:~$ export HADOOP_CONF_DIR=/home/ubuntu
ubuntu@server1:~$ export PIG_CLASSPATH=/home/ubuntu
ubuntu@server1:~$ pig -x mapreduce -f test.pig
The script ran and returned correct result but in the console I found a lot of "LocalJobRunner" and in MapReduce job tracker web interface no job is reported. 该脚本已运行并返回了正确的结果,但是在控制台中我发现了很多“ LocalJobRunner”,并且在MapReduce作业跟踪器Web界面中未报告任何作业。 Can any one tell me why doesn't it run in mapreduce mode and why doesn't it report any error of that situation?
谁能告诉我为什么它不以mapreduce模式运行,为什么它不报告这种情况的任何错误? How can I run it in mapreduce mode?
如何在mapreduce模式下运行它?
My cluster is 4.1.3 (freshly installed, all configuration is default) with Pig 0.10.0-cdh4.1.3. 我的集群是Pig 1.30.0-cdh4.1.3(全新安装,默认为所有配置)。
2013-02-20 18:57:15,843 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.0-cdh4.1.3 (rexported) compiled Jan 26 2013, 17:35:45
2013-02-20 18:57:15,843 [main] INFO org.apache.pig.Main - Logging error messages to: /home/ubuntu/epi-tre/pig_1361386635821.log
2013-02-20 18:57:16,519 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://server1:8020
2013-02-20 18:57:16,524 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:17,249 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:17,826 [main] WARN org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:17,827 [main] WARN org.apache.hadoop.conf.Configuration - dfs.permissions.supergroup is deprecated. Instead, use dfs.permissions.superusergroup
2013-02-20 18:57:17,827 [main] WARN org.apache.hadoop.conf.Configuration - dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects
2013-02-20 18:57:17,827 [main] WARN org.apache.hadoop.conf.Configuration - dfs.replication.interval is deprecated. Instead, use dfs.namenode.replication.interval
2013-02-20 18:57:17,827 [main] WARN org.apache.hadoop.conf.Configuration - dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir
2013-02-20 18:57:17,827 [main] WARN org.apache.hadoop.conf.Configuration - dfs.access.time.precision is deprecated. Instead, use dfs.namenode.accesstime.precision
2013-02-20 18:57:17,827 [main] WARN org.apache.hadoop.conf.Configuration - dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
2013-02-20 18:57:17,827 [main] WARN org.apache.hadoop.conf.Configuration - fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
2013-02-20 18:57:17,828 [main] WARN org.apache.hadoop.conf.Configuration - dfs.http.address is deprecated. Instead, use dfs.namenode.http-address
2013-02-20 18:57:17,828 [main] WARN org.apache.hadoop.conf.Configuration - dfs.replication.considerLoad is deprecated. Instead, use dfs.namenode.replication.considerLoad
2013-02-20 18:57:17,828 [main] WARN org.apache.hadoop.conf.Configuration - dfs.write.packet.size is deprecated. Instead, use dfs.client-write-packet-size
2013-02-20 18:57:17,828 [main] WARN org.apache.hadoop.conf.Configuration - dfs.permissions is deprecated. Instead, use dfs.permissions.enabled
2013-02-20 18:57:17,828 [main] WARN org.apache.hadoop.conf.Configuration - dfs.block.size is deprecated. Instead, use dfs.blocksize
2013-02-20 18:57:17,828 [main] WARN org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
2013-02-20 18:57:17,828 [main] WARN org.apache.hadoop.conf.Configuration - dfs.name.dir.restore is deprecated. Instead, use dfs.namenode.name.dir.restore
2013-02-20 18:57:17,828 [main] WARN org.apache.hadoop.conf.Configuration - dfs.https.need.client.auth is deprecated. Instead, use dfs.client.https.need-auth
2013-02-20 18:57:17,829 [main] WARN org.apache.hadoop.conf.Configuration - topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl
2013-02-20 18:57:17,829 [main] WARN org.apache.hadoop.conf.Configuration - dfs.backup.http.address is deprecated. Instead, use dfs.namenode.backup.http-address
2013-02-20 18:57:17,829 [main] WARN org.apache.hadoop.conf.Configuration - dfs.secondary.http.address is deprecated. Instead, use dfs.namenode.secondary.http-address
2013-02-20 18:57:17,829 [main] WARN org.apache.hadoop.conf.Configuration - dfs.safemode.extension is deprecated. Instead, use dfs.namenode.safemode.extension
2013-02-20 18:57:17,829 [main] WARN org.apache.hadoop.conf.Configuration - dfs.df.interval is deprecated. Instead, use fs.df.interval
2013-02-20 18:57:17,829 [main] WARN org.apache.hadoop.conf.Configuration - fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
2013-02-20 18:57:17,829 [main] WARN org.apache.hadoop.conf.Configuration - dfs.https.client.keystore.resource is deprecated. Instead, use dfs.client.https.keystore.resource
2013-02-20 18:57:17,830 [main] WARN org.apache.hadoop.conf.Configuration - dfs.datanode.max.xcievers is deprecated. Instead, use dfs.datanode.max.transfer.threads
2013-02-20 18:57:17,830 [main] WARN org.apache.hadoop.conf.Configuration - dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address
2013-02-20 18:57:17,830 [main] WARN org.apache.hadoop.conf.Configuration - topology.script.number.args is deprecated. Instead, use net.topology.script.number.args
2013-02-20 18:57:17,830 [main] WARN org.apache.hadoop.conf.Configuration - dfs.balance.bandwidthPerSec is deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec
2013-02-20 18:57:17,830 [main] WARN org.apache.hadoop.conf.Configuration - dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
2013-02-20 18:57:17,830 [main] WARN org.apache.hadoop.conf.Configuration - dfs.safemode.threshold.pct is deprecated. Instead, use dfs.namenode.safemode.threshold-pct
2013-02-20 18:57:17,830 [main] WARN org.apache.hadoop.conf.Configuration - dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir
2013-02-20 18:57:17,830 [main] WARN org.apache.hadoop.conf.Configuration - fs.checkpoint.period is deprecated. Instead, use dfs.namenode.checkpoint.period
2013-02-20 18:57:17,830 [main] WARN org.apache.hadoop.conf.Configuration - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2013-02-20 18:57:18,102 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY
2013-02-20 18:57:18,460 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-02-20 18:57:18,475 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2013-02-20 18:57:18,507 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2013-02-20 18:57:18,508 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2013-02-20 18:57:18,530 [main] WARN org.apache.hadoop.conf.Configuration - session.id is deprecated. Instead, use dfs.metrics.session-id
2013-02-20 18:57:18,531 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
2013-02-20 18:57:18,553 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-02-20 18:57:18,563 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-02-20 18:57:18,566 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2458140745309154844.jar
2013-02-20 18:57:22,423 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2458140745309154844.jar created
2013-02-20 18:57:22,453 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-02-20 18:57:22,544 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-02-20 18:57:22,553 [Thread-4] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2013-02-20 18:57:22,581 [Thread-4] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2013-02-20 18:57:22,735 [Thread-4] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:22,735 [Thread-4] WARN org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:22,790 [Thread-4] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-02-20 18:57:22,790 [Thread-4] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2013-02-20 18:57:22,851 [Thread-4] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2013-02-20 18:57:23,045 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-02-20 18:57:23,055 [Thread-4] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:23,057 [Thread-4] WARN org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:23,141 [Thread-5] INFO org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter set in config null
2013-02-20 18:57:23,178 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.df.interval is deprecated. Instead, use fs.df.interval
2013-02-20 18:57:23,178 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects
2013-02-20 18:57:23,187 [Thread-5] WARN org.apache.hadoop.conf.Configuration - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2013-02-20 18:57:23,187 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir
2013-02-20 18:57:23,188 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir
2013-02-20 18:57:23,188 [Thread-5] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:23,188 [Thread-5] WARN org.apache.hadoop.conf.Configuration - fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
2013-02-20 18:57:23,188 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.block.size is deprecated. Instead, use dfs.blocksize
2013-02-20 18:57:23,188 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.access.time.precision is deprecated. Instead, use dfs.namenode.accesstime.precision
2013-02-20 18:57:23,188 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
2013-02-20 18:57:23,188 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
2013-02-20 18:57:23,188 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.replication.considerLoad is deprecated. Instead, use dfs.namenode.replication.considerLoad
2013-02-20 18:57:23,189 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.balance.bandwidthPerSec is deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec
2013-02-20 18:57:23,189 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.safemode.threshold.pct is deprecated. Instead, use dfs.namenode.safemode.threshold-pct
2013-02-20 18:57:23,189 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.http.address is deprecated. Instead, use dfs.namenode.http-address
2013-02-20 18:57:23,189 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.name.dir.restore is deprecated. Instead, use dfs.namenode.name.dir.restore
2013-02-20 18:57:23,189 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.https.client.keystore.resource is deprecated. Instead, use dfs.client.https.keystore.resource
2013-02-20 18:57:23,189 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address
2013-02-20 18:57:23,189 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.backup.http.address is deprecated. Instead, use dfs.namenode.backup.http-address
2013-02-20 18:57:23,190 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.permissions is deprecated. Instead, use dfs.permissions.enabled
2013-02-20 18:57:23,190 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.safemode.extension is deprecated. Instead, use dfs.namenode.safemode.extension
2013-02-20 18:57:23,190 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.datanode.max.xcievers is deprecated. Instead, use dfs.datanode.max.transfer.threads
2013-02-20 18:57:23,190 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.https.need.client.auth is deprecated. Instead, use dfs.client.https.need-auth
2013-02-20 18:57:23,190 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
2013-02-20 18:57:23,190 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.replication.interval is deprecated. Instead, use dfs.namenode.replication.interval
2013-02-20 18:57:23,190 [Thread-5] WARN org.apache.hadoop.conf.Configuration - fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
2013-02-20 18:57:23,190 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.write.packet.size is deprecated. Instead, use dfs.client-write-packet-size
2013-02-20 18:57:23,190 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.permissions.supergroup is deprecated. Instead, use dfs.permissions.superusergroup
2013-02-20 18:57:23,191 [Thread-5] WARN org.apache.hadoop.conf.Configuration - topology.script.number.args is deprecated. Instead, use net.topology.script.number.args
2013-02-20 18:57:23,191 [Thread-5] WARN org.apache.hadoop.conf.Configuration - dfs.secondary.http.address is deprecated. Instead, use dfs.namenode.secondary.http-address
2013-02-20 18:57:23,191 [Thread-5] WARN org.apache.hadoop.conf.Configuration - fs.checkpoint.period is deprecated. Instead, use dfs.namenode.checkpoint.period
2013-02-20 18:57:23,191 [Thread-5] WARN org.apache.hadoop.conf.Configuration - topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl
2013-02-20 18:57:23,191 [Thread-5] WARN org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:23,193 [Thread-5] INFO org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter is org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter
2013-02-20 18:57:23,246 [Thread-5] WARN mapreduce.Counters - Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-02-20 18:57:23,333 [Thread-5] INFO org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
2013-02-20 18:57:23,344 [Thread-5] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3bfc47
2013-02-20 18:57:23,362 [Thread-5] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed hdfs://server1:8020/tmp/cf_rating.10000.csv:0+676168
2013-02-20 18:57:23,374 [Thread-5] INFO org.apache.hadoop.mapred.MapTask - io.sort.mb = 100
2013-02-20 18:57:23,507 [Thread-5] INFO org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720
2013-02-20 18:57:23,507 [Thread-5] INFO org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680
2013-02-20 18:57:23,639 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
2013-02-20 18:57:24,485 [Thread-5] INFO org.apache.hadoop.mapred.LocalJobRunner -
2013-02-20 18:57:24,488 [Thread-5] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output
2013-02-20 18:57:25,059 [Thread-5] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0
2013-02-20 18:57:25,065 [Thread-5] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
2013-02-20 18:57:25,087 [Thread-5] INFO org.apache.hadoop.mapred.LocalJobRunner -
2013-02-20 18:57:25,098 [Thread-5] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
2013-02-20 18:57:25,104 [Thread-5] WARN mapreduce.Counters - Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-02-20 18:57:25,130 [Thread-5] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@461d318f
2013-02-20 18:57:25,130 [Thread-5] INFO org.apache.hadoop.mapred.LocalJobRunner -
2013-02-20 18:57:25,137 [Thread-5] INFO org.apache.hadoop.mapred.Merger - Merging 1 sorted segments
2013-02-20 18:57:25,164 [Thread-5] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 25 bytes
2013-02-20 18:57:25,164 [Thread-5] INFO org.apache.hadoop.mapred.LocalJobRunner -
2013-02-20 18:57:25,336 [Thread-5] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
2013-02-20 18:57:25,337 [Thread-5] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:25,338 [Thread-5] WARN org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:25,339 [Thread-5] INFO org.apache.hadoop.mapred.LocalJobRunner -
2013-02-20 18:57:25,341 [Thread-5] INFO org.apache.hadoop.mapred.Task - Task attempt_local_0001_r_000000_0 is allowed to commit now
2013-02-20 18:57:25,371 [Thread-5] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://server1:8020/tmp/temp-1313079537/tmp188373172
2013-02-20 18:57:25,371 [Thread-5] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
2013-02-20 18:57:25,373 [Thread-5] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_r_000000_0' done.
2013-02-20 18:57:28,144 [main] WARN org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for job job_local_0001
2013-02-20 18:57:28,147 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-02-20 18:57:28,151 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.0.0-cdh4.1.3 0.10.0-cdh4.1.3 ubuntu 2013-02-20 18:57:18 2013-02-20 18:57:28 GROUP_BY
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTimeAvgReduceTime Alias Feature Outputs
job_local_0001 1 1 n/a n/a n/a n/a n/a n/a a,b,c GROUP_BY,COMBINER hdfs://server1:8020/tmp/temp-1313079537/tmp188373172,
Input(s):
Successfully read 0 records from: "/tmp/cf_rating.10000.csv"
Output(s):
Successfully stored 0 records in: "hdfs://server1:8020/tmp/temp-1313079537/tmp188373172"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_local_0001
2013-02-20 18:57:28,151 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2013-02-20 18:57:28,158 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-02-20 18:57:28,159 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(10000)
原来是我忘了部署MapReduce“客户端配置”,所以我的整个集群都以本地模式运行:“>已解决。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.