I tried to run mapreduce by following this tutorial.
I uploaded the files mrjob.conf, readme.txt and word_count.py on EC2 instance in the folder ~/hello_mapreduce and tryed to run the command:
python word_count.py -r emr README.txt
which returned the following:
no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
Traceback (most recent call last):
File "word_count.py", line 21, in <module>
MRWordFrequencyCount.run()
File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 494, in run
mr_job.execute()
File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 512, in execute
super(MRJob, self).execute()
File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 147, in execute
self.run_job()
File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 207, in run_job
with self.make_runner() as runner:
File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 535, in make_runner
return super(MRJob, self).make_runner()
File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 162, in make_runner
return EMRJobRunner(**self.emr_job_runner_kwargs())
File "/usr/local/lib/python2.7/dist-packages/mrjob/emr.py", line 579, in __init__
self._fix_s3_scratch_and_log_uri_opts()
File "/usr/local/lib/python2.7/dist-packages/mrjob/emr.py", line 664, in _fix_s3_scratch_and_log_uri_opts
s3_conn = self.make_s3_conn()
File "/usr/local/lib/python2.7/dist-packages/mrjob/fs/s3.py", line 226, in make_s3_conn
host=self._s3_endpoint)
File "/usr/lib/python2.7/dist-packages/boto/__init__.py", line 108, in connect_s3
return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)
File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 154, in __init__
suppress_consec_slashes=suppress_consec_slashes)
File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 486, in __init__
host, config, self.provider, self._required_auth_capability())
File "/usr/lib/python2.7/dist-packages/boto/auth.py", line 406, in get_auth_handler
'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials
I might be wrong but I guess that the possible/main problem is that the file mrjob.conf is not visible but don't know why.
The content of mrjob.conf is the following:
runners:
emr:
aws_access_key_id: xxxxx
aws_secret_access_key: xxxxxxxxxx
ec2_key_pair: EMR
ec2_key_pair_file: ~/.ssh/EMR.pem # ~/ and $ENV_VARS allowed here
ssh_tunnel_to_job_tracker: true
As an AWS user newbie I would be very thankful if anyone could help to solve the issue in order to successfully run mapreduce using mrjob and AWS.
Thank you!
Solved.
I ran the job with the command ( explicitly defined location ):
python word_count.py -r emr readme.txt --conf-path mrjob.conf
and the lines to establish the ssh connections were removed from the .conf file:
runners:
emr:
aws_access_key_id: xxxxx
aws_secret_access_key: xxxxxxxxxx
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.