Performing datastore backup to cloud storage

Question

I am attempting to create a scheduled backup of datastore via my Python Flask application (Python 3) to cloud storage. I am comfortable with the scheduling aspect of it however am having difficulty with the export.

I was using https://cloud.google.com/datastore/docs/schedule-export as a starting point however it references

from google.appengine.api import urlfetch

which is no longer supported. I have been looking into urllib

import urllib.request

url = 'https://datastore.googleapis.com/v1/projects/application-name-placeholder'
timestamp = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
output_url_prefix = 'gs://datastore-backup-test-name-placeholder/example'


query = client.query(kind='__kind__')
query.keys_only()
kinds = [entity.key.id_or_name for entity in query.fetch()]

query = client.query(kind='__namespace__')
query.keys_only()
all_namespaces = [entity.key.id_or_name for entity in query.fetch()]

entity_filter = {
    'kinds': kinds,
    'namespace_ids': all_namespaces
}
request = {
    'project_id': 'application-name-placeholder',
    'output_url_prefix': output_url_prefix,
    'entity_filter': entity_filter
}

headers = {
    'Content-Type': 'application/json'
}
response = urllib.request.Request(url)
response.add_header('Content-type', 'application/json')
result = urllib.request.urlopen(response, data=bytes(json.dumps(request), encoding="utf-8"))

At the moment I am getting

urllib.error.HTTPError: HTTP Error 404: Not Found

Not sure if my url for datastore is the correct approach but think there are other issues with my approach. Some guidance would be appreciated.

Answer 1

In your URL you have url = 'https://datastore.googleapis.com/v1/projects/application-name-placeholder' , but linked documentation has url = 'https://datastore.googleapis.com/v1/projects/%s:export' % app_id . You are missing the trailing :export.

Given that you are trying to export your whole database, you should remove your entity filter. Without an entity filter the managed export will export your entire database.