How to log custom information in the BigQuery data access logs?

Question

I have set up a log export for the BigQuery data access logs as per this link .

In my use case, clients don't have permission to access the BigQuery directly. They call REST API which fetches the data from BigQuery and serves it to clients. I need to log some custom information, like user ID which requested the data etc., along with the BigQuery data access logs. I tried using the userAgent field by setting the application name with the user ID but it is not showing up in the data access logs.

Is there a way to log custom information in the data access logs?

Answer 1

Below are some of the options I see.

---

1 - use jobReference.jobId job's property

You can set it such that it will consists of whatever info you have available at a time of jobs API call

jobReference.jobId  string  [Required] The ID of the job. The ID must contain   
                            only letters (a-z, A-Z), numbers (0-9),   
                            underscores (_), or dashes (-).   
                            The maximum length is 1,024 characters.  

I think 1024 characters are more than enough to encode whatever metadata you need

---

2 - Another option is to use configuration.labels property of job.

configuration.labels object [Experimental] The labels associated with this job.
                            You can use these to organize and group your jobs.   
                            Label keys and values can be no longer than 63  
                            characters, can only contain lowercase letters, numeric
                            characters, underscores and dashes. International 
                            characters are allowed. Label values are optional. 
                            Label keys must start with a letter and each label in 
                            the list must have a different key.

You can read more about labeling at Labeling Datasets . Even though it describes Labels for Datasets - it is exactly same concept for use with Jobs

---

Finally, I would recommend avoid adjusting user's query - especially for the purpose of inserting some metadata

Answer 2

Can you add custom data as a comment on the queries?

 # hello, I'm a comment and I have meta-data
 SELECT * FROM `wherever`

Just be careful with the first line, as you can use it to request BigQuery a choice between #legacySQL and #standardSQL

 #standardSQL
 # for metadata, respect the first line of the query to choose SQL variant
 # hello, I'm a comment and I have meta-data
 SELECT * FROM `wherever`