How to copy datastore entities between namespaces

Question

I want to copy all the data in one namespace, say www.mysite.com, to another namespace, say nightly.latest.mysite.appspot.com. What's the best way to do this?

The example namespaces are not random: they're the namespaces that are set by a NamespaceFilter for the given domains that serve the app.

I want to be able to pull all the 'production' data into a 'non-production' namespace for testing.

Answer 1

Namespace is the part of the Key. So you can't change or copy all data from one namespace to another. As I understand all you can do is to fetch all objects from one namespace and create NEW objects with the same properties in another namespace.

Answer 2

I'm using appengine-mapreduce for this. I won't walk through setting it up in detail. You can read the getting started guides for that information.

Right now you have to enter each class name to be copied. TODO is figuring out how to loop over all the __Stat_Kind__ results programmatically so each kind/class doesn't have to be specified separately.

import java.util.logging.Logger;

import org.apache.hadoop.io.NullWritable;

import com.google.appengine.api.NamespaceManager;
import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
import com.google.appengine.api.datastore.Key;
import com.google.appengine.api.datastore.KeyFactory;
import com.google.appengine.tools.mapreduce.AppEngineMapper;

public class DatastoreCopyMapper extends
        AppEngineMapper<Key, Entity, NullWritable, NullWritable> {

    private static final Logger log = Logger
            .getLogger(DatastoreCopyMapper.class.getName());
    private static String destination;

    public DatastoreCopyMapper() {
    }

    @Override
    public void taskSetup(Context context) {
        log.warning("Doing per-task setup");
        destination = context.getConfiguration().get("destination");
        log.warning("destination: " + destination);
    }

    @Override
    public void map(Key key, Entity value, Context context) {

        NamespaceManager.set(destination);
        String name = key.getName();
        long id = key.getId();
        Key destinationKey = null;
        if (name != null) {
            destinationKey = KeyFactory.createKey(key.getKind(), name);
        } else if (id != 0) {
            destinationKey = KeyFactory.createKey(key.getKind(), id);
        }
        Entity destinationEntity = new Entity(destinationKey);
        destinationEntity.setPropertiesFrom(value);
        DatastoreService datastore = DatastoreServiceFactory
                .getDatastoreService();

        datastore.put(destinationEntity);

    }
}

mapreduce.xml

<configurations>
  <configuration name="Copy between namespaces">
    <property>
      <name>mapreduce.map.class</name>
      <value>com.mysite.server.DatastoreCopyMapper</value>
    </property>
    <property>
      <name>mapreduce.inputformat.class</name>
      <value>com.google.appengine.tools.mapreduce.DatastoreInputFormat</value>
    </property>
    <property>
      <name human="Entity Kind to Map Over">mapreduce.mapper.inputformat.datastoreinputformat.entitykind</name>
      <value template="optional">User</value>
    </property>

    <property>
      <name human="Destination Namespace">mapreduce.mapper.inputformat.datastoreinputformat.destination</name>
      <value template="optional">dev.mysite.com</value>
    </property>

  </configuration>
</configurations>

Answer 3

Unfortunately there is currently no automated way to do this.

You are going to have to loop over all of your data and re-save it to the new namespace. You can read about Java and namespaces, in the multitenancy docs.