如何在本地开发服务器上启用 Google Cloud Storage？

Question

I want to use a GCS bucket as the backing for my blobstore but I cannot figure out how to set one up on my development server.

There are instructions for doing this using the developers console on the live server, but I can't find anything about how to do it on my local development machine...

Answer 1

Turns out you don't need to perform any setup at all. I just assumed there was one with particular name when uploading using the blobstore and one was created for me automatically.

Incidentally, it does not seem to be documented anywhere how you can browse files in the storage of the development server. You can do it by selecting the __GsFileInfo__ entity in the Datastore Viewer admin access to your local dev server.

Answer 2

For those trying to get Google Cloud Storage to work at all from their local Java development app server, I thought one more answer would be helpful. I managed to get my local dev app server working with non-local Google Cloud Storage, but only be digging through the code and figuring out what was needed - there doesn't appear to be documentation on this.

The goal will be to get this block of code to work locally, which reads a file from GCS:

    GcsService gcsService =
       GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
    int fileSize = (int) gcsService.getMetadata(gcsFilename).getLength();
    ByteBuffer byteBuffer = ByteBuffer.allocate(fileSize);
    GcsInputChannel inputChannel = gcsService.openReadChannel(gcsFilename, 0);
    int readResult = inputChannel.read(byteBuffer);
    byte[] fileBytes = byteBuffer.array();

If you try to do this locally, you won't find any files you've upload to GCS, because it will be trying to use a fake local GCS. Unfortunately, I haven't found a good way to upload to this local GCS, so that isn't very useful (there is no file explorer for it as there is in the cloud version, and gsutil doesn't work for it). So instead, we're going to get it to work with non-local (cloud) GCS when running in the local dev app server.

To do that, note that GcsService is created in com.google.appengine.tools.cloudstorage.GcsServiceFactory by this block of code:

if (location == SystemProperty.Environment.Value.Production || hasCustomAccessTokenProvider()) {
  rawGcsService = OauthRawGcsServiceFactory.createOauthRawGcsService(builder.build());
} else if (location == SystemProperty.Environment.Value.Development) {
  rawGcsService = LocalRawGcsServiceFactory.createLocalRawGcsService();

The above says that you need to specify a custom access token provider to get the non-local service, which you do by defining a system property. For an app engine app, you can do that in appengine-web.xml like this:

<system-properties>
    <property name="gcs_access_token_provider" value="com.mypackage.MyAccessTokenProvider" />
</system-properties>

This value of that property is a class you define that implements com.google.appengine.tools.cloudstorage.oauth.AccessTokenProvider, which provides the access token for your app. That class needs to create a GoogleCredential, which can be used to fetch an access token, using the instructions for "Other" for a GoogleCredential on https://developers.google.com/identity/protocols/OAuth2ServiceAccount#authorizingrequests .

Now it will create an OAuth GcsService that talks to the cloud, and you don't need to use the fake local storage.

Answer 3

You need to download and integrate the Google Cloud Storage Client Library for App Engine.

This library provides you the GcsService which is similar to the BlobstoreService , so you can write a file, read a file, delete a file and other functions provided from Cloud Storage

When you use the code in the development environment, the uploaded files are stored in the appengine-generated folder and an __GsFileInfo__ entity is created in the local datastore which preserves the metadata attached to the file

This library works online too, so your code will work for both development and production environments.

Here you can find the Getting Started guide and the full JavaDoc reference

Answer 4

Figured this could use some more recent findings. For me, the local cloud storage doesn't work at all. It fails with an error saying GsFileInfo is a protected entity. It works fine if you don't fire up the local datastore emulator (meaning it uses the remote cloud storage).

This seems to jive with google's latest docs: https://cloud.google.com/appengine/docs/standard/java/using-cloud-storage#using_cloud_storage_with_the_local_development_server

The App Engine local development server doesn't emulate Cloud Storage, so all Cloud Storage requests must be sent over the Internet to an actual Cloud Storage bucket.

Thus the answer from @learnj12 is the best approach. However the GoogleCredential has been deprecated, and it's unclear how to implement com.google.appengine.tools.cloudstorage.oauth.AccessTokenProvider.

Here's what worked for me:

import com.google.appengine.api.appidentity.AppIdentityService;
import com.google.appengine.api.appidentity.AppIdentityServiceFactory;
import com.google.appengine.tools.cloudstorage.oauth.AccessTokenProvider;

public class GCSTokenProvider implements AccessTokenProvider {
  @Override
  public AppIdentityService.GetAccessTokenResult getNewAccessToken(List<String> list) {
    return AppIdentityServiceFactory.getAppIdentityService().getAccessToken(Collections.singleton("cloud-platform"));
  }
}

NB. I'm using a service account, and am logged in with gcloud .