No module named boto Google Cloud Storage on Django with App Engine

Question

Right now I am working on setting up a simple API system on one of my web projects for writing to and reading from Google Cloud Storage (mostly for photo storage) yet I'm running into issues with some of the plugins while emulating in the App Engine environment.

I might start first by explaining how I got to this point. Initially, I looked right into the cloudstorage API provided by Google and attempted to use an extension which did something like this for saving files:

def _save(self, name, content):
    filename = self.location + "/" + name  # e.g. https://storage.googleapis.com/bucket/name
    filename = os.path.normpath(filename)
    filetype, encoding = mimetypes.guess_type(name)
    gss_file = gcs.open(
        filename,
        mode='w',
        content_type=filetype,
        options={
            'x-goog-acl': 'public-read',
            'cache-control': settings.GOOGLE_CLOUD_STORAGE_DEFAULT_CACHE_CONTROL,
        }
    )
    content.open()
    gss_file.write(content.read())
    content.close()
    gss_file.close()
    return name

Using this code, I was able to achieve what seemed like success. I got HTTP status codes of 200 / 201 for the POST / PUT which seemed good, but upon looking further I could see that the files did not actually upload correctly. After tinkering for a bit, I determined that it must just be because I need authorization for write permissions and did not provide them -- likely I am getting success codes because I'm achieving a response, it just isn't the response I want (ie yes this page exists but there is no file there) so I went looking for a way to authenticate.

I took a look right into the python tutorial provided by Google here but did not have much luck with this one at all. Where I am at right now is that I have installed boto and gcs_oauth2_boto_plugin via pip , and have also obtained a client ID and client secret from the API Manager Credentials in App Engine. Though upon trying to run a page while using the following setup:

#!/usr/bin/python

import boto
import gcs_oauth2_boto_plugin


GOOGLE_STORAGE = 'gs'

CLIENT_ID = '(my client id)'
CLIENT_SECRET = '(my client secret)'
gcs_oauth2_boto_plugin.SetFallbackClientIdAndSecret(CLIENT_ID, CLIENT_SECRET)

BUCKET = '(my bucket name)'


def save_blob(directory, name, content):
    dst_uri = boto.storage_uri(BUCKET + '/' + directory + name, GOOGLE_STORAGE)
    dst_uri.new_key().set_contents_from_file(content)
    print 'Successfully created "%s/%s"' % (dst_uri.bucket_name, dst_uri.object_name)

With the above code, upon trying to run a web page where I implement this, I receive the following error:

ImportError: no module named boto.

In the past, when I've run into this type of error, it's been one of a few things:

1. I haven't installed the module on pip (I have)
2. I haven't added it to the library section in app.yaml (This generates an error "library boto is not supported" anyway)
3. I haven't added it to the PYTHONPATH (I can import it no problem from Python on the command line no problem)

I tried adding these modules to my lib directory before, but it was all a mess of undefined imports, and I'm not sure if that was necessary anyway. I'm going to try doing that again now, but I'm not sure if that's what the answer is here. I've even tried adding the modules to the INSTALLED_APPS in my settings, but that did not solve any problems either. I'm really a bit stuck here, and would appreciate anyone's help with this. Thanks!

EDIT: It may help to note that I am trying to use the XML API based on the tutorial supplied by Google here . Should I be trying the JSON API instead? From reading a few other SO posts, it seemed like the general consensus was that the JSON API did not allow me as much flexibility in providing additional backend logic. Is this true? Or should I be trying the JSON API? At this point I'm quite unsure of the difference between the two since I have not been able to get either of them to work and have no experience with either. Any extra information on that is also appreciated.

Answer 1

With help from @voscausa I tried some of the things I had attempted previously, but instead this time from within App Engine using the primary project with which I hold the Cloud Storage bucket. With this, I was able to write files with no problem. What I will probably do from here is just set up an API which I can use from my other projects to read and write files through the main project instead of trying to do it directly from my other projects. Thanks @voscausa for pointing me in the right direction! It really was this easy all along. I was just trying to make it harder for myself I suppose.