Getting Anonymous caller does not have storage.objects.create access error while running wordcount on dataflow

Question

I am running the dataflow quickstart of wordcount module in apache beam package using python. I was able to run it locally on my machine. But, when I try to run it on dataflow service by specifying dataflow runner in the runner field, it is throwing error as:

 raise exceptions.HttpError.FromResponse(http_response)
RuntimeError: apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/hogwarts/o?alt=json&name=results%2Fbeam-temp-outputs-acb95dc89b0511eb9507185e0fbdf17c%2Fc5994362-a8ca-45cc-a375-85fafb619329.outputs&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ABg5-UzsW55q9DUiXsssbR3c0I_X85oFxjAo-FFn_ZeZ3HtZdlWPXjcd1-euaquMiOw4znDrIl8qYkBeIOBUhUlBibI', 'date': 'Sun, 11 Apr 2021 20:37:04 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '600', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'alt-svc': 'h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'status': '401'}>, content <{
  "error": {
    "code": 401,
    "message": "Anonymous caller does not have storage.objects.create access to hogwarts/results/beam-temp-outputs-acb95dc89b0511eb9507185e0fbdf17c/c5994362-a8ca-45cc-a375-85fafb619329.outputs.",
    "errors": [
      {
        "message": "Anonymous caller does not have storage.objects.create access to hogwarts/results/beam-temp-outputs-acb95dc89b0511eb9507185e0fbdf17c/c5994362-a8ca-45cc-a375-85fafb619329.outputs.",
        "domain": "global",
        "reason": "required",
        "locationType": "header",
        "location": "Authorization"
      }
    ]
  }
}

I get this error when I try to run this code in google cloud sdk:-

python -m apache_beam.examples.wordcount \ --region $REGION \ --input gs://dataflow-samples/shakespeare/kinglear.txt \ --output gs://hogwarts/results/outputs \--runner DataflowRunner \ --project MyProject1 \ --temp_location gs://hogwarts/tmp/ 

Can you please tell me how to fix this issue. I have already provided the required access & I also updated my GOOGLE APPLICATION CREDENTIALS with respective keys.

Answer 1

You need to grant the storage.objects.create on the gs://hogwarts to the compute service account. You can find this in the IAM section in GCP console.

The compute account looks like this: xxxxxxxx-compute@developer.gserviceaccount.com , where xxxxxxx is your project number.