I have a Google bucket with the following tree ( Note the folder named "/"! ):
"bucket-1"
|
|--- "data.csv"
|
|--- "/"
|
|--- "runs"
|
|--- "run-1"
| |
| |--- "data.csv"
|
|--- "run-2"
|
|--- "data.csv"
I want to access the objects (.csv files) using the Python library libcloud in the sub-folder "/".
I can access data.csv
which is outside of the "/" folder:
>>> client.get_object(container_name='bucket-1', object_name='/data.csv')
<Object: name=/data.csv, size=181580, hash=8252d90d95b7b1cb7b4e699b90fbcce3, provider=Google Cloud Storage ...>
Using gsutil with two slashes I can see objects in "/":
>>> gsutil ls "gs://bucket-1//runs/run-1"
gs://bucket-1//runs/run-1/data.csv
However with libcloud if I do client.get_object(container_name='bucket-1', object_name='//runs/run-1/data.csv')
or client.get_object(container_name='bucket-1', object_name='/runs/run-1/data.csv')
I get the error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/andrey/miniconda3/envs/mostly-cloud/lib/python3.6/site-packages/libcloud/storage/drivers/s3.py", line 342, in get_object
object_name=object_name)
libcloud.storage.types.ObjectDoesNotExistError: <ObjectDoesNotExistError in <libcloud.storage.drivers.google_storage.GoogleStorageDriver object at 0x7f40560cd4e0>, value=None, object = //runs/run-1/data.csv>
On the other hand,
client.list_container_objects(client.get_container("bucket-1"))
[<Object: name=/runs/run-1/data.csv, size=357683, hash=..., provider=Google Cloud Storage ...>, <Object: name=/runs/run-2/data.csv, size=357683, hash=..., provider=Google Cloud Storage ...>]
So, how to get an object located in the "/" directory?
I reproduced your scenario in order to test this behavior. I was able to create this hierarchy through:
gsutil cp your-file gs://your-bucket//abc
This is a weird behavior and it shouldn't be allowed.
If you try to create a folder with this name from the GCP console, you will see the message:
Forward slashes (/) are not allowed in folder names.
For that reason I created a Public Issue Tracker where you can get feedback regarding this issue.
About the naming of your folders, you can take a look at Google's documentation about how subdirectories are working.
To sum up, you shouldn't be allowed to create a folder with this name. The best course of action now is to avoid names like this and prefer a string name which you will be able to handle.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.