[英]GCP - Message from PubSub to BigQuery
I need to get the data from my pubsub message and insert into bigquery.我需要从我的 pubsub 消息中获取数据并插入到 bigquery 中。
What I have:我有的:
const topicName = "-----topic-name-----";
const data = JSON.stringify({ foo: "bar" });
// Imports the Google Cloud client library
const { PubSub } = require("@google-cloud/pubsub");
// Creates a client; cache this for further use
const pubSubClient = new PubSub();
async function publishMessageWithCustomAttributes() {
// Publishes the message as a string, e.g. "Hello, world!" or JSON.stringify(someObject)
const dataBuffer = Buffer.from(data);
// Add two custom attributes, origin and username, to the message
const customAttributes = {
origin: "nodejs-sample",
username: "gcp",
};
const messageId = await pubSubClient
.topic(topicName)
.publish(dataBuffer, customAttributes);
console.log(`Message ${messageId} published.`);
}
publishMessageWithCustomAttributes().catch(console.error);
I need to get the data/attributes from this message and query in BigQuery, anyone can help me?我需要从此消息中获取数据/属性并在 BigQuery 中查询,有人可以帮助我吗?
Thaks in advance!提前谢谢!
In fact, there is 2 solutions to consume the messages: either a message per message, or in bulk.事实上,有两种消费消息的解决方案:每条消息一条消息,或者批量。
Firstly, before going in detail, and because you will perform BigQuery calls (or Facebook API calls), you will spend a lot of the processing time to wait the API response.首先,在详细介绍之前,由于您将执行 BigQuery 调用(或 Facebook API 调用),您将花费大量处理时间来等待 API 响应。
Create Cloud Function on the topic , or a Push Subscription to call a HTTP triggered Cloud Functions在主题上创建 Cloud Function或推送订阅以调用HTTP 触发的 Cloud Functions
This solution offers a better message throughput processing (you can process more than 250 message per Cloud Run service), but don't have a real advantage because you are limited by the API call latency.此解决方案提供了更好的消息吞吐量处理(每个 Cloud Run 服务可以处理超过 250 条消息),但没有真正的优势,因为您受到 API 调用延迟的限制。
EDIT 1编辑 1
Code sample代码示例
// For pubsunb triggered function
exports.logMessageTopic = (message, context) => {
console.log("Message Content")
console.log(Buffer.from(message.data, 'base64').toString())
console.log("Attribute list")
for (let key in message.attributes) {
console.log(key + " -> " + message.attributes[key]);
};
};
// For push subscription
exports.logMessagePush = (req, res) => {
console.log("Message Content")
console.log(Buffer.from(req.body.message.data, 'base64').toString())
console.log("Attribute list")
for (let key in req.body.message.attributes) {
console.log(key + " -> " + req.body.message.attributes[key]);
};
};
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.