how to move data between datasets in different regions?

Question

I'm using BigQuery integrated with Firebase and all the datasets are in the same Project. My analytics dataset is in useast-4 but for some reason my firebase_imported_segments dataset region is just marked as US I'd like to move data from the analytics dataset into a table in the firebase_imported_segments .

At first, I tried a simple INSERT query but I get the error firebase_imported_segments was not found in location us-east4

So then I tried building a SELECT statement and exporting the rows using "Save Results > Big Query Table" but that gives a similar error that the destination dataset is not found. Oddly enough, if I create a table in firebase_imported_segments and try to save the results using that table name, I get a "Table already exists" error. So it's not that it can't find the firebase_imported_segments dataset, it just won't create a new table in that dataset.

How can I get around this? I saw some BQ documentation that moving data between regions is possible but I didn't a simple walkthrough of how it's accomplished. I'm also confused by why firebase would put some data in one specific region ( useast-4 ) and then other data in a multi-region ( US ) if they aren't compatible.

Answer 1

You can move datasets using "Copy" in the BigQuery UI then delete the old dataset. See Copy dataset documentation .

Option 1: Use the Copy button.

1. Go to the BigQuery page in the Cloud console.

2. In the Explorer panel, expand your project and select a dataset.

3. Expand the More Actions option (triple dot button) and click Open.

4. Click Copy. In the Copy dataset dialog that appears, do the following:

   a. In the Dataset field, either create a new dataset or select an existing dataset ID from the list.

   Dataset names within a project must be unique. The project and dataset can be in different regions, but not all regions are supported for cross-region dataset copying.

   b. In the Location field, the location of the source dataset is displayed.

   c. Optional: To overwrite both data and schema of the destination tables with the source tables, select the Overwrite destination tables checkbox.

   d. To copy the dataset, click Copy.

To avoid additional storage costs, consider deleting the old dataset.

Option 2: Use the BigQuery Data Transfer Service.

1. Enable the BigQuery Data Transfer Service.

2. Create a transfer for your data source.

I tested this and can confirm that it works. I created a dataset in us-east4 named analytics_us_regional and has a table named east_4_table and copied it to a dataset located in US .

Copy us-east4 to US dataset:

When copy is initiated a data transfer job is created:

Copied to US :

With regards to the data in firebase located in us-east4 based from the firebase export to BQ . When the export is enabled the first time, the user will define the location of the tables. It might be possible that us-east4 region was selected initially.

Answer 2

Don't know if it will work in your case, but I had a dataset in europe-west1 and I want to copy it to EU region, I have done these two ways and it both worked:

First way : 1- Click on the dataset you want to copy and click on "COPY".

2- On the copy menu on the dataset destination click on "CREATE NEW DATA SET" and select the destination region you want that dataset to be. Click on CREATE DATA SET.

3 - On the "Copy data set" menu click on COPY.

4 - You will get an error "Cannot create a transfer in REGION_EUROPE_WEST_1 when destination dataset is located in JURISDICTION_EU" but a dataset with no tables will be created on your destination Region.

5 - Now if you try to copy the source dataset by clicking on COPY and selecting the dataset created in set 4, it will work now.

Second way : (best way) 1 - Open a New Query sheet Click on MORE- >Query settings-> Advanced options, uncheck the "Automatic location selection" and select the destination region or Multi-region you want (in my case EU).

2- On this query sheet run "CREATE SCHEMA your_new_dataset_name " -> this will create the dataset "your_new_dataset_name" in the destination region selected in point 1.

3 - Click on the dataset you want to copy and click on "COPY".

4 - On the copy menu on the Data Set destination select the dataset created in point 2, and click on COPY.

Both ways under the wood utilize the BigQuery Data Transfer Service but you don't need to access the service directly. In fact, both ways do exactly the same thing which is creating a destination empty dataset in the correct region you want to copy yours, once you have that the Copy function will work correctly.