GCP App Engine - Could not load the default credentials

Question

I'm working on a project, that uses GCP and App Engine. In dev it will print out errors saying:

2020-09-20 15:07:24 dev[development]  Error: Could not load the default credentials. Browse 
to https://cloud.google.com/docs/authentication/getting-started for more information.      
at GoogleAuth.getApplicationDefaultAsync (/workspace/node_modules/google-auth-
library/build/src/auth/googleauth.js:161:19)      at runMicrotasks (<anonymous>)      at 
processTicksAndRejections (internal/process/task_queues.js:97:5)      at runNextTicks 
(internal/process/task_queues.js:66:3)      at listOnTimeout (internal/timers.js:518:9)      
at processTimers (internal/timers.js:492:7)      at async GoogleAuth.getClient 
(/workspace/node_modules/google-auth-library/build/src/auth/googleauth.js:503:17)      at 
async GrpcClient._getCredentials (/workspace/node_modules/google-
gax/build/src/grpc.js:108:24)      at async GrpcClient.createStub 
(/workspace/node_modules/google-gax/build/src/grpc.js:229:23)

Keep in mind this is development mode, but is running on the GCP App Engine infrastructure, it is not being run on localhost. I'm viewing the logs with the command:

gcloud app logs tail -s dev

According to the GCP App Engine docs @ https://cloud.google.com/docs/authentication/production#cloud-console

If your application runs inside a Google Cloud environment that has a default service 
account, your application can retrieve the service account credentials to call Google Cloud 
APIs.

I checked my app engine service accounts. And I have a default service account and it is active. Please see the redacted image here:

So I guess my question is: If I have an active default service account, and my application is supposed to automatically use the default service account key when it makes API calls, why am I seeing this authentication error? What am I doing wrong?

Edit: here's the code that is printing out errors:

async function updateCrawledOnDateForLink (crawlRequestKey: EntityKey, linkKey: EntityKey): Promise<void> {

  try {
        
    const datastore = new Datastore();
    
    const crawlRequest = await datastore.get(crawlRequestKey)
    
    const brand = crawlRequest[0].brand.replace(/\s/g, "")
    
    await data.Link.update(
      +linkKey.id,
      { crawledOn: new Date() },
      null,
      brand)
        
  } catch (error) {
        console.error('updateCrawledOnDateForLink ERROR:', `CrawlRequest: ${crawlRequestKey.id}`, `Link: ${linkKey.id}`, error.message)
  }
}

My prediction is that creating a new Datastore() each time is the problem, but let me know your thoughts.

Answer 1

The fact that removing new Datastore() from a couple of functions solves the issue indicates that the issue is not with Authentication with App Engine but it is with Datastore, which confirms the documentation piece you shared.

I believe that the issue is that Datastore is getting lost with the credentials when you are creating multiple instances of it in your code for some unknown reason.

Since you mentioned in the comments that you don't really need multiple instances of Datastore in your code the solution to your problem is to use a single instance in a global variable and use that variable in multiple functions.