如何使Firebase数据库与BigQuery保持同步？

Question

We are working on a project where a lot of data is involved. Now we recently read about Google BigQuery. But how can we export the data to this platform? We have seen the sample of importing logs into Google BigQuery. But this does not contain information about updating and deleting data (only inserting).

So our objects are able to update their data. And we have a limited amount of queries on the BigQuery tables. How can we synchronize our data without exceeding the BigQuery quota limits.

Our current function code:

'use strict';

// Default imports.

const functions = require('firebase-functions');
const bigQuery = require('@google-cloud/bigquery')();

// If you want to change the nodes to listen to REMEMBER TO change the constants below.
// The 'id' field is AUTOMATICALLY added to the values, so you CANNOT add it.

const ROOT_NODE = 'categories';
const VALUES = [
    'name'
];

// This function listens to the supplied root node.
// When the root node is completed empty all of the Google BigQuery rows will be removed.
// This function should only activate when the root node is deleted.

exports.root = functions.database.ref(ROOT_NODE).onWrite(event => {
    if (event.data.exists()) {
        return;
    }

    return bigQuery.query({
        query: [
            'DELETE FROM `stampwallet.' + ROOT_NODE + '`',
            'WHERE true'
        ].join(' '),
        params: []
    });
});

// This function listens to the supplied root node, but on child added/removed/changed.
// When an object is inserted/deleted/updated the appropriate action will be taken.

exports.children = functions.database.ref(ROOT_NODE + '/{id}').onWrite(event => {
    const id = event.params.id;

    if (!event.data.exists()) {
        return bigQuery.query({
            query: [
                'DELETE FROM `stampwallet.' + ROOT_NODE + '`',
                'WHERE id = ?'
            ].join(' '),
            params: [
                id
            ]
        });
    }

    const item = event.data.val();

    if (event.data.previous.exists()) {
        let update = [];
        for (let index = 0; index < VALUES.length; index++) {
            const value = VALUES[index];

            update.push(item[value]);
        }
        update.push(id);

        return bigQuery.query({
            query: [
                'UPDATE `stampwallet.' + ROOT_NODE + '`',
                'SET ' + VALUES.join(' = ?, ') + ' = ?',
                'WHERE id = ?'
            ].join(' '),
            params: update
        });
    }

    let template = [];
    for (let index = 0; index < VALUES.length; index++) {
        template.push('?');
    }

    let create = [];
    create.push(id);
    for (let index = 0; index < VALUES.length; index++) {
        const value = VALUES[index];

        create.push(item[value]);
    }

    return bigQuery.query({
        query: [
            'INSERT INTO `stampwallet.' + ROOT_NODE + '` (id, ' + VALUES.join(', ') + ')',
            'VALUES (?, ' + template.join(', ') + ')'
        ].join(' '),
        params: create
    });
});

What would be the best way to sync firebase to bigquery?

Answer 1

BigQuery supports UPDATE and DELETE, but not frequent ones - BigQuery is an analytical database, not a transactional one.

To synchronize a transactional database with BigQuery you can use approaches like:

• Export a daily dump, and import it into BigQuery.
• Treat updates and deletes as new events, and keep appending events to your BigQuery event log.
• Use a tool like https://github.com/MemedDev/mysql-to-google-bigquery .
• Approaches like "BigQuery at WePay part III: Automating MySQL exports every 15 minutes with Airflow , and dealing with updates"

With Firebase you could schedule a daily load to BigQuery from their daily backups:

• https://firebase.googleblog.com/2016/10/announcing-automated-daily-backups-for-the-firebase-database.html

Answer 2

... way to sync firebase to bigquery?

I recommend considering streaming all you data into BigQuery as a historical data. You can mark entries as new(insert), update or delete. Then, on BigQuery side, you can write query that will resolve most recent values for specific record based on whatever logic you have.
So your code can be reused almost 100% - just fix logic of UPDATE / DELETE to have it as INSERT

// When an object is inserted/deleted/updated the appropriate action will be taken .

So our objects are able to update their data. And we have a limited amount of queries on the BigQuery tables. How can we synchronize our data without exceeding the BigQuery quota limits?

Yes, BigQuery supports UPDATE , DELETE , INSERT as a part of Data Manipulation Language .
General availability was announced in BigQuery Standard SQL at March 8, 2017

Before considering using this feature for syncing BigQuery with transactional data – please take a look at Quotas , Pricing and Known Issues .

Below are some excerpts!

Quotas (excerpts)
DML statements are significantly more expensive to process than SELECT statements.
• Maximum UPDATE/DELETE statements per day per table: 96
• Maximum UPDATE/DELETE statements per day per project: 1,000

Pricing (excerpts, extra highlighting + comment added)
BigQuery charges for DML queries based on the number of bytes processed by the query.
The number of bytes processed is calculated as follows:

UPDATE Bytes processed = sum of bytes in referenced fields in the scanned tables + the sum of bytes for all fields in the updated table at the time the UPDATE starts.
DELETE Bytes processed = sum of bytes of referenced fields in the scanned tables + sum of bytes for all fields in the modified table at the time the DELETE starts.

Comment by post author: As you can see you will be charged for whole table scan even though you update just one row! This is a key here for decision making, I think!

Known Issues (excerpts)
• DML statements cannot be used to modify tables with REQUIRED fields in their schema.
• Each DML statement initiates an implicit transaction, which means that changes made by the statement are automatically committed at the end of each successful DML statement. There is no support for multi-statement transactions.
• The following combinations of DML statements are allowed to run concurrently on a table:

• UPDATE and INSERT
• DELETE and INSERT

• INSERT and INSERT

  Otherwise one of the DML statements will be aborted.
  For example, if two UPDATE statements execute simultaneously against the table then only one of them will succeed.

• Tables that have been written to recently via BigQuery Streaming (tabledata.insertall) cannot be modified using UPDATE or DELETE statements. To check if the table has a streaming buffer, check the tables.get response for a section named streamingBuffer. If it is absent, the table can be modified using UPDATE or DELETE statements.

Answer 3

The reason why you didn't find update and delete functions in BigQuery is they are not supported by BigQuery. BigQuery has only append and truncate operations. If you want to update or delete row in your BigQuery you'll need to delete the whole database and write it again with modified row or without it. It is not a good idea.

BigQuery is used to store big amounts of data and have a quick access to it, for example it is good for collecting data from different sensors. But for your customer database you need to use MySQL or NoSQL database.