Google Cloud Datastore，如何查询更多结果

Question

Straight and simple, I have the following function, using Google Cloud Datastore Node.js API:

fetchAll(query, result=[], queryCursor=null) {
  this.debug(`datastoreService.fetchAll, queryCursor=${queryCursor}`);
  if (queryCursor !== null) {
    query.start(queryCursor);
  }
  return this.datastore.runQuery(query)
  .then( (results) => {
    result=result.concat(results[0]);
    if (results[1].moreResults === _datastore.NO_MORE_RESULTS) {
      return result;
    } else {
      this.debug(`results[1] = `, results[1]);
      this.debug(`fetch next with queryCursor=${results[1].endCursor}`);
      return this.fetchAll(query, result, results[1].endCursor);
    }
  });
}

The Datastore API object is in the variable this.datastore ;

The goal of this function is to fetch all results for a given query, notwithstanding any limits on the number of items returned per single runQuery call.

I have not yet found out about any definite hard limits imposed by the Datastore API on this, and the documentation seems somewhat opaque on this point, but I only noticed that I always get results[1] = { moreResults: 'MORE_RESULTS_AFTER_LIMIT' } , indicating that there are still more results to be fetched, and the results[1].endCursor remains stuck on constant value that is passed on again on each iteration.

So, given some simple query that I plug into this function, I just go on running the query iteratively, setting the query start cursor (by doing query.start(queryCursor); ) to the endCursor obtained in the result of the previous query. And my hope is, obviously, to obtain the next bunch of results on each successive query in this iteration. But I always get the same value for results[1].endCursor . My question is: Why?

Conceptually, I cannot see a difference to this example given in the Google Documentation :

// By default, google-cloud-node will automatically paginate through all of
// the results that match a query. However, this sample implements manual
// pagination using limits and cursor tokens.
function runPageQuery (pageCursor) {
  let query = datastore.createQuery('Task')
    .limit(pageSize);

  if (pageCursor) {
    query = query.start(pageCursor);
  }

  return datastore.runQuery(query)
    .then((results) => {
      const entities = results[0];
      const info = results[1];

      if (info.moreResults !== Datastore.NO_MORE_RESULTS) {
        // If there are more results to retrieve, the end cursor is
        // automatically set on `info`. To get this value directly, access
        // the `endCursor` property.
        return runPageQuery(info.endCursor)
          .then((results) => {
            // Concatenate entities
            results[0] = entities.concat(results[0]);
            return results;
          });
      }

      return [entities, info];
    });
}

(except for the fact, that I don't specify a limit on the size of the query result by myself, which I have also tried, by setting it to 1000, which does not change anything.)

Why does my code run into this infinite loop, stuck on each step at the same "endCursor"? And how do I correct this?

Also, what is the hard limit on the number of results obtained per call of datastore.runQuery() ? I have not found this information in the Google Datastore documentation thus far.

Thanks.

Answer 1

Looking at the API documentation for the Node.js client library for Datastore there is a section on that page titled "Paginating Records" that may help you. Here's a direct copy of the code snippet from the section:

var express = require('express');
var app = express();

var NUM_RESULTS_PER_PAGE = 15;

app.get('/contacts', function(req, res) {
  var query = datastore.createQuery('Contacts')
    .limit(NUM_RESULTS_PER_PAGE);

  if (req.query.nextPageCursor) {
    query.start(req.query.nextPageCursor);
  }

  datastore.runQuery(query, function(err, entities, info) {
    if (err) {
      // Error handling omitted.
      return;
    }

    // Respond to the front end with the contacts and the cursoring token
    // from the query we just ran.
    var frontEndResponse = {
      contacts: entities
    };

    // Check if  more results may exist.
    if (info.moreResults !== datastore.NO_MORE_RESULTS) {
      frontEndResponse.nextPageCursor = info.endCursor;
    }

    res.render('contacts', frontEndResponse);
  });
});

Maybe you can try using one of the other syntax options (instead of Promises). The runQuery method can take a callback function as an argument, and that callback's parameters include explicit references to the entities array and the info object (which has the endCursor as a property).

And there are limits and quotas imposed on calls to the Datastore API as well. Here are links to official documentation that address them in detail:

Limits

Quotas