How to get expired table data in bigquery, If the expired time is more than two days?

Question

I have a process in which I get the table data in bigquery on the daily basis. I need some old table data but unfortunately they're expired now and their expiration time is more than two days. I know we can get back the table data if it's deleted and deleted time is less than two days, but is it possible in the case of expired table and the time is more than 2 days?

I tried using timestamp of 2 days back and tried to get it using bq tool, but I need data which was deleted 2 days before.

Answer 1

Bigquery tables don't necessarily expire in 2 days. You can set them to whatever you like: https://cloud.google.com/bigquery/docs/managing-tables#updating_a_tables_expiration_time

Once they expired, there's no way to retrieve the table, unless you have snapshots in place. In that case, you can restore a snapshot and use that to get the data you want. See this SO question on how to do that: How can I undelete a BigQuery table?

Answer 2

GCP Support here!

Actually if you read through the SO question linked by @niczky12 and as stated in the documentation :

It's possible to restore a table within 2 days of deletion. By leveraging snapshot decorator functionality, you may be able to reference a table prior to the deletion event and then copy it. Note the following:

• You cannot reference a deleted table if you have already created a new table with the same name in the same dataset.

• You cannot reference a deleted table if you deleted the dataset that housed the table, and you have already created a new dataset with the same name.

At this point, unfortunately it is impossible to restore the deleted data.

Answer 3

To add for future searchers here. I was able to follow the explanation below on medium and restore data that was still there 7 days ago.

Actually the Cloud Shell in the UI gave back the max time to go back when i tried a date that was too far int he future. The max time they gave back was 7 days in EPOCH Miliseconds. Just type that in the convertor below and add 1 or 2 hours and you should be good. Don't take the exact copy of what the console provides, as that is outdated by the time it's printed.

• https://medium.com/@dhafnar/how-to-instantly-recover-a-table-in-google-bigquery-544a9b7e7a8d
• https://www.epochconverter.com/

And make sure to set future tables to never delete! (or a date you know). This can be found in the table details, and also on dataset level in console.cloud environment for bigquery.

Answer 4

(as of 2022 at least) In general, you can recover data BQ tables for 7 days via time-travel. See GCP doc:

• https://cloud.google.com/bigquery/docs/time-travel

and the related: How can I undelete a BigQuery table?