使用Toil连接到AWS Cloud中的S3存储桶时禁止使用403

Question

I am a newbie in Toil and AWS trying to run HelloWorld.py example in the Toil Document. I have already successfully installed toil and related python packages on my local mac laptop and have setup my account at AWS. I have created a small leader/worker cluster

$ cgcloud create-cluster toil -s 2 -t m3.large 
and started it:
$ cgcloud ssh toil-leader 

This changed my screen prompt to:

mesosbox@ip-172-31-25-135:~$

Then from an other window on my mac, I started the Toil HellowWorld example with with command:

$ python2.7 HelloWorld.py --batchSystem=mesos --mesosMaster=mesos-master:5050 aws:us-west-2:my-aws-jobstore

And I got the following output:

Apples-Air 2017-06-02 19:30:53,524 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
    Apples-Air 2017-06-02 19:30:53,524 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
    Apples-Air 2017-06-02 19:30:54,852 MainThread WARNING toil.jobStores.aws.jobStore: Exception during panic
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 209, in initialize
        self.destroy()
      File "/usr/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 1334, in destroy
        self._bind(create=False, block=False)
      File "/usr/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 241, in _bind
        versioning=True)
      File "/usr/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 721, in _bindBucket
        bucket = self.s3.get_bucket(bucket_name, validate=True)
      File "/usr/local/lib/python2.7/site-packages/boto/s3/connection.py", line 502, in get_bucket
        return self.head_bucket(bucket_name, headers=headers)
      File "/usr/local/lib/python2.7/site-packages/boto/s3/connection.py", line 535, in head_bucket
        raise err
    S3ResponseError: S3ResponseError: 403 Forbidden

    Traceback (most recent call last):
      File "helloWorld.py", line 22, in <module>
        print(Job.Runner.startToil(j, options)) #Prints Hello, world!, ….
      File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 740, in startToil
        with Toil(options) as toil:
      File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 614, in __enter__
        jobStore.initialize(config)
      File "/usr/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 209, in initialize
        self.destroy()
      File "/usr/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 206, in initialize
        self._bind(create=True)
      File "/usr/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 241, in _bind
        versioning=True)
      File "/usr/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 721, in _bindBucket
        bucket = self.s3.get_bucket(bucket_name, validate=True)
      File "/usr/local/lib/python2.7/site-packages/boto/s3/connection.py", line 502, in get_bucket
        return self.head_bucket(bucket_name, headers=headers)
      File "/usr/local/lib/python2.7/site-packages/boto/s3/connection.py", line 535, in head_bucket
        raise err
    boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

Please help.
Thanks.
---John

Answer 1

I realize that this answer is a little late. One problem I notice is with the mesosMaster argument.

Instead, your command should have look like

python2.7 HelloWorld.py --batchSystem=mesos --mesosMaster=172.31.25.135:5050 aws:us-west-2:my-aws-jobstore

Notice that I replaces mesos-master with the actual IP address from

mesosbox@ip-172-31-25-135:~$

Hopefully in the future, one will not need to pass this argument at all, however this is not yet implemented as of 26 July 2017.

Also for further problems with Toil you will probably have better luck posting a new issue to the Toil Github page.