How to authenticate BigQuery in R on Linux Server (bigrquery)

Question

We are trying to authenticate to BigQuery in an R script scheduled to run on a linux server. We have:

bigrquery::bq_auth(path = '~/path-to/credentials/our-creds.json')

and when I run this locally in R, I get this prompt

The bigrquery package is requesting access to your Google account.
Select a pre-authorised account or enter '0' to obtain a new token.
Press Esc/Ctrl + C to cancel.

1: myemail@gmail.com

Selection: 

When we run this on our linux server via Rscript our_script.R , we get the following error:

Error: Can't get Google credentials.
Are you running bigrquery in a non-interactive session? Consider:
  * Call `bq_auth()` directly with all necessary specifics.
Execution halted

When we run this on our linux server via R to open up R, and then manually running the line of code, we get:

> bigrquery::bq_auth(path = '~/path-to/credentials/our-creds.json')
Waiting for authentication in browser...
Press Esc/Ctrl + C to abort

and because we don't have a browser to authenticate in (we are SSH'd into our linux server which lives on GCP), the terminal hangs here...

How can we successfully authenticate on the server, so Rscript out_script.R can run successfully?

Edit: Info on our server:

let me know if I can share additional info to help with this. Also we tried adding myemail@gmail.com to bigrquery's email parameter, but this did not resolve the issue. I see there is a token parameter and perhaps we can manually add a token in somewhere? This is all super frustrating because I thought using the service account key was supposed to resolve this issue, and yet we are pointing to a valid service account key with bigquery permissions, and yet still cannot authenticate on the server.

Answer 1

The same question was asked in the RStudio community forum. Posting the discussion here for posterity.

A few troubleshooting ideas:

... R script scheduled to run on a linux server....

Whenever I hear "scheduled", I think "cron" and when I hear "cron", I think "path problems". So in the server / schedule context, I recommend doing very pedantic checks that you truly understand what working directory is, who the user is, and whether a path is resolving to what you intend.

However, the fact that this service account token doesn't even seem to work for you in a local interactive session suggests something is wrong with the file. Historically, people have tried to use other JSON files, such as the JSON for an OAuth client, as if it were the JSON for a service account, which does not work, for obvious reasons.

You can gain more insight into how this is failing and falling through to interactive auth by using the gargle verbosity option as described here:

gargle.r-lib.org - Troubleshooting gargle auth - gargle

And in case you have not already found it (and for the benefit of other readers) here is a general overview of how to set up non-interactive auth:

gargle.r-lib.org - Non-interactive auth - gargle

As acknowledged by @Canovice in the same forum thread , the issue was due to a faulty service account key used in the bq_auth() function.