[英]Write query in BigQuery from Pubsub
need some help.需要一些帮助。
I'm receiving messages with data in a PubSub topic, I need to insert the data I get from the message and query in BigQuery with a background cloud function(PUB/SUB)...我正在接收带有 PubSub 主题中数据的消息,我需要插入从消息中获取的数据并使用后台云功能 (PUB/SUB) 在 BigQuery 中查询...
What I manage to do:我设法做的事情:
/**
* Triggered from a message on a Cloud Pub/Sub topic.
*
* @param {!Object} event Event payload.
* @param {!Object} context Metadata for the event.
*/
exports.insertBigQuery = (message, context) => {
extractQuery(message.data);
};
function extractQuery(pubSubMessage){
// Decide base64 the PubSub message
let logData = Buffer.from(pubSubMessage, 'base64').toString();
// Convert it in JSON
let logMessage= JSON.parse(logData);
console.log(logMessage.customerToken)
console.log(logMessage.fbclid)
console.log(logMessage.fbc)
console.log(logMessage.fbp)
console.log(logMessage.firstHitTS)
console.log(logMessage.consentFB)
main();
return logMessage
}
"use strict";
function main() {
const { BigQuery } = require("@google-cloud/bigquery");
const bigquery = new BigQuery();
async function query() {
const query = `INSERT INTO MYTABLE( customerToken, fbclid, fbc, fbp, firstHitTS, consentFB)
VALUES ("customerTokenSCRIPTCLOUD","fbclidSCRIPT"," fbcSCRIPTCLOUD"," fbpSCRIPTCLOUD","2021-01-05",TRUE )`;
const options = {
query: query,
location: "US",
};
const [job] = await bigquery.createQueryJob(options);
console.log(`Job ${job.id} started.`);
const [rows] = await job.getQueryResults();
console.log("Rows:");
rows.forEach((row) => console.log(row));
}
query();
}
Now every time I receive a message I query in bigQuery, but my VALUES is hard coded, as you can see here:现在,每次我收到一条消息时,我都会在 bigQuery 中查询,但我的 VALUES 是硬编码的,如您在此处看到的:
const query = `INSERT INTO devsensetestprojects.TestDataSet.fbSimpleData( customerToken, fbclid, fbc, fbp, firstHitTS, consentFB)
VALUES ("customerTokenSCRIPTCLOUD","fbclidSCRIPT"," fbcSCRIPTCLOUD"," fbpSCRIPTCLOUD","2021-01-05",TRUE )`;
What I'm not able to do is to get the values from function extractQuery(pubSubMessage)
and use them in my query the same way I use in the function (logMessage.SOMEVALUE) to have the correct values I need.我无法做的是从
function extractQuery(pubSubMessage)
中获取值,并在查询中使用它们,就像我在 function (logMessage.SOMEVALUE) 中使用的一样,以获得我需要的正确值。
Thanks in advance!提前致谢!
As you said, you are beginner in development.正如您所说,您是开发的初学者。 Here a more concise and efficient code.
这里有更简洁高效的代码。 I didn't tested it but it is closer of what you want.
我没有测试它,但它更接近你想要的。 Let me know is some part are mysterious for you!
让我知道有些部分对你来说是神秘的!
// Make them global to load them only when the Cloud Function instance is created
// They will be reused in the subsequent processing and until the instance deletion
const { BigQuery } = require("@google-cloud/bigquery");
const bigquery = new BigQuery();
exports.insertBigQuery = async (message, context) => {
// Decode base64 the PubSub message
let logData = Buffer.from(message.data, 'base64').toString();
// Convert it in JSON
let logMessage= JSON.parse(logData);
const query = createQuery(logMessage)
const options = {
query: query,
location: "US",
};
const [job] = await bigquery.createQueryJob(options);
console.log(`Job ${job.id} started.`);
// Only wait the end of the job. Theere is no row as answer, it's only an insert
await job.getQueryResults();
}
function createQuery(logMessage) {
// You maybe have to format correctly the logMessage.firstHitTS to be accepted by BigQuery as a date.
return `INSERT INTO MYTABLE(customerToken, fbclid, fbc, fbp, firstHitTS, consentFB)
VALUES (logMessage.customerToken, logMessage.fbclid, logMessage.fbc, logMessage.fbp,
logMessage.firstHitTS, logMessage.consentFB)`;
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.