HttpForbiddenError when trying to access Google Cloud Storage from Apache Beam

Question

I've trying to simple access with Apache Beam to Google Cloud storage from Compute Engine VM. Sure, I've set up default application login with command gcloud auth application-default login and add access to the storage for compute engine service account. I've run the pipeline with DirectRunner and got the Error: apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions {'gs://{THIS MY BUCKETNAME}/source/sales_transactions.csv': HttpForbiddenError()}

#import print library
import logging

#import apache beam library
import apache_beam as beam

#import pipeline options.
from apache_beam.options.pipeline_options import  PipelineOptions

#Create a pipeline
plOps = beam.Pipeline(options=PipelineOptions())

#--------------------------------------------------
# 1.Read from a text file.
#--------------------------------------------------

#Read the file from Google Cloud Storage
transactions = ( plOps 
                | 'Read Transaction CSV'
                    >> beam.io.ReadFromText('gs://{THIS MY BUCKETNAME}/data/sales_transactions.csv')
                )

printSize(transactions,'Raw Transactions')

Answer 1

I've partially resolved the issue - reading ok. I've login by root with sudo su on DEVELOPER MACHINE and Apache Beam got access to read file from GS. But when I try to write into GS bucket like this

#Write output to a text file
( custTypeCount | 'Write to GS Text'
        >> beam.io.WriteToText('gs://{MY BUCKET NAME}/output/customertype-summary.txt')
)  

script got error:

RuntimeError: HttpForbiddenError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/{MY BUCKET NAME}/o?uploadType=resumable&alt=json&name=output%2Fbeam-temp-customertype-summary.txt-7bea505ad0bf11e9b69c42010a800002%2F55a9057e-18e5-4171-9db4-9e55601b2a8d.customertype-summary.txt>: response: <{'status': '403', 'content-length': '208', 'vary': 'Origin, X-Origin', 'server': 'UploadServer', 'x-guploader-uploadid': 'AEnB2Upo4RBzVV1S51_uWhcCiK_uK_iOSRAdAb8HWMhxznaPr0JcHKWxKDLwHbtTIYvHuMjyESV4dZqAfN3TaWYMqr5gQeypcQ', 'date': 'Fri, 06 Sep 2019 16:00:44 GMT', 'content-type': 'application/json; charset=UTF-8', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"'}>, content <{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "insufficientPermissions",
    "message": "Insufficient Permission"
   }
  ],
  "code": 403,
  "message": "Insufficient Permission"
 }
}
> [while running 'Write to GS Text/Write/WriteImpl/WriteBundles']

And I haven't any idea what permission should I add

Answer 2

Yeah! I solved it. Just recreate default application credential - really I don't know what was wrong with them. I've made it unambiguously. Otherwise folder /root/.config/gcloud hadn't existed. But something happened with credentials.

So, how to solve it: 1. sudo su 2. drop folder /root/.config/gcloud 3. Run again gcloud auth application-default login

Be lucky!