[英]Consume Data from Confluent Kafka Topic and Exit using Python
[英]Consume data from KAFKA topic and extract fields from it and store in MySQL using python
我想使用以下命令使用來自 Kafka 主題的數據,如下所示:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTestTopic --from-beginning
然后這將輸出以下內容(僅粘貼前 2 行輸出,但它將是多行...):
&time=1561768216000&gameCategory=PINPOINT&category=ONE&uniqueId=2518Z-0892A-0030O-16H70&transactionType=CRD&familyId=000-222-115-11119&realTs=1561768319000&sortId=1&msg=SET-UP+PRAYER+%26+intercession+begins+in+just+30+minutes.&remoteIpAddress=127.0.0.1&userAgent=HTTP&
&uniqueId=872541806296826880&time=1571988786000&gameCategory=NOTIFY&category=TWO&transactionType=CRD&familyId=401-222-115-89387&sortId=1&realTs=1571988989000&msg=This-is+a+reminder.&remoteIpAddress=127.0.0.1&userAgent=HTTPS&
我想從輸出中使用以下內容:
房地產
家庭 ID
留言
唯一身份
並且您可以看到每個元素都由與號 ('&') 分隔。 它們並不總是在相同的索引/位置,所以我不確定是否需要正則表達式? 最終,當我在本地運行的 MySQL 上進行查詢時,我會看到:
描述測試表;
+----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+--------------+------+-----+---------+-------+
| realTs | bigint(20) | YES | | NULL | |
| familyId | varchar(255) | YES | | NULL | |
| msg | text | YES | | NULL | |
| uniqueId | varchar(255) | YES | | NULL | |
+----------+--------------+------+-----+---------+-------+
4 rows in set (0.00 sec)
SELECT * FROM testTable;
+---------------+-------------------+-----------------------------------------------------------+-------------------------+
| realTs | familyId | msg | uniqueId |
+---------------+-------------------+-----------------------------------------------------------+-------------------------+
| 1561768319000 | 000-222-115-11119 | SET-UP+PRAYER+%26+intercession+begins+in+just+30+minutes. | 2518Z-0892A-0030O-16H70 |
| 1571988989000 | 401-222-115-89387 | This-is+a+reminder. | 872541806296826880 |
+---------------+-------------------+-----------------------------------------------------------+-------------------------+
到目前為止我有什么? 我有一個帶有 python 的 mysql 連接器,我可以在其中連接到本地 mysql 等,但是我正在努力解析它並插入它......
使用 Python,您可以使用urllib.parse.parse_qs
在 Python 字典中檢索 URL 查詢字符串組件,您可以稍后迭代以在 MySQL 數據庫中插入數據。
例如:
from urllib.parse import parse_qs
line = "&time=1561768216000&gameCategory=PINPOINT&category=ONE&uniqueId=2518Z-0892A-0030O-16H70&transactionType=CRD&familyId=000-222-115-11119&realTs=1561768319000&sortId=1&msg=SET-UP+PRAYER+%26+intercession+begins+in+just+30+minutes.&remoteIpAddress=127.0.0.1&userAgent=HTTP&uniqueId=872541806296826880&time=1571988786000&gameCategory=NOTIFY&category=TWO&transactionType=CRD&familyId=401-222-115-89387&sortId=1&realTs=1571988989000&msg=This-is+a+reminder.&remoteIpAddress=127.0.0.1&userAgent=HTTPS&"
o = parse_qs(line)
print(o)
結果:
{'time': ['1561768216000', '1571988786000'], 'gameCategory': ['PINPOINT', 'NOTIFY'], 'category': ['ONE', 'TWO'], 'uniqueId': ['2518Z-0892A-0030O-16H70', '872541806296826880'], 'transactionType': ['CRD', 'CRD'], 'familyId': ['000-222-115-11119', '401-222-115-89387'], 'realTs': ['1561768319000', '1571988989000'], 'sortId': ['1', '1'], 'msg': ['SET-UP PRAYER & intercession begins in just 30 minutes.', 'This-is a reminder.'], 'remoteIpAddress': ['127.0.0.1', '127.0.0.1'], 'userAgent': ['HTTP', 'HTTPS']}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.