IBM Cloud Object Storage Connection issue Watson studio Python notebook

Question

I have been trying to get my Python notebook in Watson Studio to connect to and retrieve data from my cloud storage for hours and read many tutorials but no idea why it is not working, code below:

credentials = {
    'BUCKET': 'openai-data',
    'URL': 'https://s3.private.us-east.cloud-object-storage.appdomain.cloud',
    'SECRET_KEY': '',
    'API_KEY': '*********************', #this key has been *d out but the value is there in my code
    'RESOURCE_INSTANCE_ID': 'crn:v1:bluemix:public:cloud-object-storage:global:a/e448d36ef93f4d3ca55077db903d3461:51ce6e50-4e92-41d0-b002-5023e815cadc::',
    'FILE': 'test_Y_smallsample.h5',
    'ACCESS_KEY': ''
}


from ibm_botocore.client import Config
import ibm_boto3
cos = ibm_boto3.resource(service_name='s3',
    ibm_api_key_id=credentials['API_KEY'],
    ibm_service_instance_id=credentials['RESOURCE_INSTANCE_ID'],
    ibm_auth_endpoint='https://iam.bluemix.net/oidc/token',
    config=Config(signature_version='oauth'),
    endpoint_url=credentials['URL'])

files = cos.Bucket('openai-data').objects.all()
for file in files:
    print("Item: {0} ({1} bytes).".format(file.key, file.size))

this produces the error: CredentialRetrievalError: Error when retrieving credentials from https://iam.bluemix.net/oidc/token : HttpCode(400) - Retrieval of tokens from server failed.

same result if I use " https://iam.cloud.ibm.com/identity/token " for the ibm_auth_endpoint

I've also tried a separate connection with HMAC credentials as well, but can't find any tutorials that show you how to incorporate those either...

Please help!

Thanks

Answer 1

This can be caused due to the use of invalid apikey. To get an apikey value, go to the 'Service credentials' menu of your storage instance and then click on "View credentials" of the "WDP-Project-Management-..." and see the value in the 'apikey' field.

This issue does not seem to be caused by invalid endpoint_url, bu anyway, to get the endpoint_url, go to Buckets > 'your bucket' > Configuration, and then choose an endpoint according to you needs (Private/Public/Direct). When filling the field, use "https://".

See below a code snippet to get data from a cvs file directly:

import pandas as pd 
import ibm_boto3 
from ibm_botocore.client import Config 
cos = ibm_boto3.client(service_name='s3',
    ibm_api_key_id='<apikey>', 
    ibm_auth_endpoint="https://iam.ng.bluemix.net/oidc/token",
    config=Config(signature_version='oauth'),
    endpoint_url="<endpoint>")
obj = cos.get_object(Bucket='<bucket_name>', Key='<csv_file>')
df = pd.read_csv(obj['Body'])