python script is not running without sudo - why?

Question

This is my python script which downloads the most recent image from my S3 bucket. When I run this script using sudo python script.py it does as expected, but not when I run it as python script.py . In this case, the script finishes cleanly without exceptions or process lockup, but there is no image file.

Why does this happen? Is there something I could do at boto's library end or any other thing?

import boto
import logging


def s3_download():
    bucket_name = 'cloudadic'
    conn = boto.connect_s3('XXXXXX', 'YYYYYYY')
    bucket = conn.get_bucket(bucket_name)

    for key in bucket.list('ocr/uploads'):
        try:
            l = [(k.last_modified, k) for k in bucket]
            key = sorted(l, cmp=lambda x, y: cmp(x[0], y[0]))[-1][1]
            res = key.get_contents_to_filename(key.name)
        except:
            logging.info(key.name + ":" + "FAILED")

if __name__ == "__main__":
     s3_download()

Answer 1

Presumably the problem is that you try to store things somewhere your user doesn't have permission to. The second issue is that your script is hiding the error. The except block completely ignores what sort of exceptions occur (and of course consumes them so you never see them), and uses logging.info which by default doesn't show; that should probably be at least a warning and would be better if it showed something about what went wrong. By the way, odds are you shouldn't be posting S3 authentication keys here.

Answer 2

As @Nearoo in comments had suggested to use except Exception as inst to catch exceptions.

catching the exception like this

except Exception as inst:
            print(type(inst))
            print(inst.args)
            print(inst)

Should fetch you this error if the script is compiled using python 3x If you would compile your script using python 2.7, this error wouldn't come.

Probably you might be having multiple versions of python on your system, which should be the reason for your difference in behavior of python script.py and sudo python script.py and for the same @mootmoot answer suggests to use virtualenv

'cmp' is an invalid keyword argument for this function .

Now if you would have googled this error, you should find that cmp has been deprecated in python 3x and suggestions would be to use key instead.

add these imports

import functools
from functools import cmp_to_key

and replace with these

key2 = sorted(l, key = cmp_to_key(lambda x,y: (x[0] > y[0]) - (x[0] < y[0])))[-1][1]
res = key2.get_contents_to_filename(key2.name)

(x[0] > y[0]) - (x[0] < y[0]) is an alternate to cmp(x[0], y[0])

cmp is replaced by key and cmp_to_key is used from functools lib

Check this out

Answer 3

Always install virtual environment before you start development in python.

The issue you face is typical python newbie problem : use sudo to install all the pypi package.

When you do sudo pip install boto3 , it will install pypi package to system workspace and only accessible by sudo . In such case, sudo python script.py will works, because it has the rights to access the package.

To resolve this issues and isolation of development environment (so you don't pollute different project with differnt pypi package), python developer will install python virtual environment (from above link), then use mkvirtualenv to create your project workspace, run pip install and python setup.py install to install required package to the environment, then you can run python without the sudo python .

Python virtualenv is also deployed inside production environment for the same reason.

Important Note : avoid boto and boto2. AWS no longer support them nor with bug fixing(AWS is not officially support boto2, use it at your own risk). Switch to boto3.

---

For the exceptional handling issues, @Yann Vernier has mentioned it. And exception error will not logged by logging.info . You may try with logging.debug or simple use raise to raise the actual exception error.