ORM for google cloud datastore

Question

https://developers.google.com/datastore/docs/overview

It looks like datastore in GAE but without ORM (object relation model). May I used the same ORM model as datastore on GAE for Cloud Datastore? or Is there any ORM support can be found for Cloud Datastore?

Answer 1

App Engine Datastore high level APIs, both first party (db, ndb) and third party (objectify, slim3), are built on top of low level APIs:

• datastore_rpc for Python
• DatastoreService / AsyncDatastoreService for Java

Replacing the App Engine specific versions of these interfaces/classes to work on top of The Google Cloud Datastore API will allow you to use these high level APIs outside of App Engine.

The high level API code itself should not have to change (much).

Answer 2

Google Cloud Datastore only provides a low-level API (proto and json ) to send datastore RPCs.

NDB and similar higher level libraries could be adapted to use a lower level wrapper like googledatastore ( reference ) instead of google.appengine.datastore.datastore_rpc

Answer 3

For the .NET folks out there, I just created pogo, which is a .NET ORM that supports saving POCOs to Google Cloud Datastore, and querying them using LINQ.

It is available on nuget, called "pogo", and the source is hosted here - http://code.thecodeprose.com/pogo

For example, here's an insert and a lookup:

var unique = Guid.NewGuid().ToString();

var poco = new TestDepartment
{
    Id = unique,
    Code = 123,
    Director = new TestEmployee { FirstName = "Boss" }
};

using (var session = _datastore.OpenSession())
{
    session.Store(poco);
    session.SaveChanges();
    var lookupPoco = session.Load<TestDepartment>(unique).SingleOrDefault();

    Assert.AreEqual("Boss", lookupPoco.Director.FirstName);
}

and here's a query:

using (var session = _datastore.OpenSession())
{
    var results = session.Query<TestEmployee>()
          .Where(t => t.HourlyRate > 50.0)
          .ToList();

    Assert.IsTrue(results.Count > 0);
}

Answer 4

This is not an exact answer and We already know Google is working for NDB Library , I couldn't wait for that.

What I tried is to write NDB Properties not listed in datastore_v1_pb2.py , such as GeoPt .

class GCDFoo(ndb.Model):
    latlng = ndb.GeoPtProperty()

in this case, if we read the entity by GCD lowlevel API, returns like following.

name: "latlng"
value {
  entity_value {
    property {
      name: "x"
      value {
        double_value: 10.0
        indexed: false
      }
    }
    property {
      name: "y"
      value {
        double_value: 10.0
        indexed: false
      }
    }
  }
  meaning: 9
}

hm, I don't really know what does 'meaning' mean, but It was important to describe GeoPt. and now We can write GeoPt property something like this.

def make_geopt_value(lat,lng):
  entity = datastore.Entity()
  prop = entity.property.add()
  datastore.helper.set_property(prop,'x',lng,indexed=False)
  prop = entity.property.add()
  datastore.helper.set_property(prop,'y',lat,indexed=False)
  value = datastore.Value()
  datastore.helper.set_value(value,entity)
  value.meaning = 9
  return value

It worked for me, still don't know if it's right approach though. Anyway I hope my answer is helpful for someone who can't wait NDB Library.

Answer 5

You can take a look on this one, this one is written in Typescript

Description: ts-datastore-orm targets to provide a strong typed and structural ORM feature for Datastore (Firestore in Datastore mode).

https://www.npmjs.com/package/ts-datastore-orm

import {BaseEntity, Batcher, Column, datastoreOrm, Entity, Transaction, DatastoreOrmDatastoreError, DatastoreOrmError} from "ts-datastore-orm";

@Entity({namespace: "testing", kind: "user", compositeIndexes: [{id: "desc"}, {string: "asc", ["object.name"]: "asc", __ancestor__: false}]})
export class User extends BaseEntity {
    @Column({generateId: true})
    public id: number = 0;

    @Column()
    public date: Date = new Date();

    @Column({index: true})
    public string: string = "";

    @Column()
    public number: number = 10;

    @Column()
    public buffer: Buffer = Buffer.alloc(1);

    @Column()
    public array: number[] = [];

    @Column({excludeFromIndexes: ["object.name"]})
    public object: any = {};

    @Column()
    public undefined: undefined = undefined;

    @Column()
    public null: null = null;
}


async function queryExamples() {
    const [user1, requestResponse1] = await User.query().runOnce();
    const [userList1, requestResponse2] = await User.findMany([1, 2, 3, 4, 5]);
    const [userList2, requestResponse3] = await User.findMany({ancestor: user1, ids: [1, 2, 3, 4, 5]});
    const [userList3, requestResponse4] = await User.query().run();

    const user2 = User.create({id: 1});
    const query = TaskGroup.query()
        .filter("number", "=", 10)
        .filterAny("anyColumn.name", ">", 5)
        .setAncestor(user2)
        .groupByAny("anyColumn.name")
        .orderAny("anyColumn.name", {descending: true})
        .offset(5)
        .limit(10);

    while (query.hasNextPage()) {
        const [entities] = await query.run();
    }

    // stream
    const stream = query.runStream()
        .on("data", (entity) => {
            //
        })
        .on("info", (info) => {
            //
        })
        .on("error", (error) => {
            //
        })
        .on("end", () => {
            //
        });
}

Answer 6

Yes. Not only can you use the same ORM models, Google Cloud Datastore allows you to read and write from your current App Engine apps storage.

https://developers.google.com/datastore/docs/activate?hl=en#google_cloud_datastore_for_an_existing_app_engine_application