Migrating data from Google Datastore (GAE) to Google Cloud SQL

Question

I am using Google Datastore for my application and now for reporting purposes i want to migrate my data from GAE Datastore to Google Cloud SQL.

What options do i have? (i have above 100k entries and doing it in deffered job is taking alot of time).

Answer 1

Reading your question, it seems you have a two-part problem: 1) Do a one-time export of all past data to Cloud SQL, 2) setup some sort of periodic ETL process to keep the two databases synced up.

For the one-time dump, based on reading the docs, it seems you have a few options here:

1. Use Bulkloader to export all of your data to a CSV and then either load it into a local MySQL instance, generate a mysqldump and follow this tutorial to load it into Cloud SQL or create a script that would connect to the Cloud SQL instance and load the data one transaction at a time.
2. Programmatically create a process inside your application to publish a task to write a given piece of data into the Cloud SQL instance, and a separate worker that would connect to the Cloud SQL db and write the data.

Personally, I'd go the bulkloader/local mysql/mysqldump route, as you are not adding any throw away code into your main application. If anything goes wrong, you can just clear the Cloud SQL instance and restart.

For the ETL piece, I'd recommend either a task queue route, as you will be able to get to near real-time, but have the ability to control costs by throttling the queue execution speed.

EDIT: This article might be interesting - covers data migration from GAE to another DB: http://www-cs-students.stanford.edu/~silver/gae.html (Meaningful content is after the rant part...)