[英]Can't Consume JSON Messages From Kafka Using Kafka-Python's Deserializer
我試圖通過Kafka發送一個非常簡單的JSON對象,並使用Python和kafka-python將其讀出來。 但是,我一直看到以下錯誤:
2017-04-07 10:28:52,030.30.9998989105:kafka.future:8228:ERROR:10620:Error processing callback
Traceback (most recent call last):
File "C:\Anaconda2\lib\site-packages\kafka\future.py", line 79, in _call_backs
f(value)
File "C:\Anaconda2\lib\site-packages\kafka\consumer\fetcher.py", line 760, in _handle_fetch_response
unpacked = list(self._unpack_message_set(tp, messages))
File "C:\Anaconda2\lib\site-packages\kafka\consumer\fetcher.py", line 539, in _unpack_message_set
tp.topic, msg.value)
File "C:\Anaconda2\lib\site-packages\kafka\consumer\fetcher.py", line 570, in _deserialize
return f(bytes_)
File "C:\Users\myUser\workspace\PythonKafkaTest\src\example.py", line 55, in <lambda>
value_deserializer=lambda m: json.loads(m).decode('utf-8'))
File "C:\Anaconda2\lib\json\__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "C:\Anaconda2\lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Anaconda2\lib\json\decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
我做了一些研究,這個錯誤最常見的原因是JSON錯了。 我在發送之前嘗試打印出JSON,方法是將以下內容添加到我的代碼中,並且JSON打印沒有錯誤。
while True:
json_obj1 = json.dumps({"dataObjectID": "test1"})
print json_obj1
producer.send('my-topic', {"dataObjectID": "test1"})
producer.send('my-topic', {"dataObjectID": "test2"})
time.sleep(1)
這讓我懷疑我可以生產json,但不會消耗它。
這是我的代碼:
import threading
import logging
import time
import json
from kafka import KafkaConsumer, KafkaProducer
class Producer(threading.Thread):
daemon = True
def run(self):
producer = KafkaProducer(bootstrap_servers='localhost:9092',
value_serializer=lambda v: json.dumps(v).encode('utf-8'))
while True:
producer.send('my-topic', {"dataObjectID": "test1"})
producer.send('my-topic', {"dataObjectID": "test2"})
time.sleep(1)
class Consumer(threading.Thread):
daemon = True
def run(self):
consumer = KafkaConsumer(bootstrap_servers='localhost:9092',
auto_offset_reset='earliest',
value_deserializer=lambda m: json.loads(m).decode('utf-8'))
consumer.subscribe(['my-topic'])
for message in consumer:
print (message)
def main():
threads = [
Producer(),
Consumer()
]
for t in threads:
t.start()
time.sleep(10)
if __name__ == "__main__":
logging.basicConfig(
format='%(asctime)s.%(msecs)s:%(name)s:%(thread)d:' +
'%(levelname)s:%(process)d:%(message)s',
level=logging.INFO
)
main()
如果刪除value_serializer和value_deserializer,我可以成功發送和接收字符串。 當我運行該代碼時,我可以看到我發送的JSON。這是一個簡短的片段:
ConsumerRecord(topic=u'my-topic', partition=0, offset=5742, timestamp=None, timestamp_type=None, key=None, value='{"dataObjectID": "test1"}', checksum=-1301891455, serialized_key_size=-1, serialized_value_size=25)
ConsumerRecord(topic=u'my-topic', partition=0, offset=5743, timestamp=None, timestamp_type=None, key=None, value='{"dataObjectID": "test2"}', checksum=-1340077864, serialized_key_size=-1, serialized_value_size=25)
ConsumerRecord(topic=u'my-topic', partition=0, offset=5744, timestamp=None, timestamp_type=None, key=None, value='test', checksum=1495943047, serialized_key_size=-1, serialized_value_size=4)
ConsumerRecord(topic=u'my-topic', partition=0, offset=5745, timestamp=None, timestamp_type=None, key=None, value='\xc2Hello, stranger!', checksum=-1090450220, serialized_key_size=-1, serialized_value_size=17)
ConsumerRecord(topic=u'my-topic', partition=0, offset=5746, timestamp=None, timestamp_type=None, key=None, value='test', checksum=1495943047, serialized_key_size=-1, serialized_value_size=4)
ConsumerRecord(topic=u'my-topic', partition=0, offset=5747, timestamp=None, timestamp_type=None, key=None, value='\xc2Hello, stranger!', checksum=-1090450220, serialized_key_size=-1, serialized_value_size=17)
所以我嘗試從使用者中刪除value_deserializer,並且該代碼執行但沒有反序列化器,消息以String形式出現,這不是我需要的。 那么,為什么value_deserializer不起作用呢? 是否有不同的方法從我應該使用的Kafka消息中獲取JSON?
首先將消息解碼為utf-8,然后json.load / dump它解決了我的問題:
value_deserializer=lambda m: json.loads(m.decode('utf-8'))
代替:
value_deserializer=lambda m: json.loads(m).decode('utf-8')
希望這對制片人也有效
事實證明問題是value_deserializer=lambda m: json.loads(m).decode('utf-8')
的解碼部分,當我將其更改為value_deserializer=lambda m: json.loads(m)
然后我看到從Kafka讀取的對象類型現在是一個字典。 基於python的JSON文檔中的以下信息是正確的:
|---------------------|------------------|
| JSON | Python |
|---------------------|------------------|
| object | dict |
|---------------------|------------------|
| array | list |
|---------------------|------------------|
| string | unicode |
|---------------------|------------------|
| number (int) | int, long |
|---------------------|------------------|
| number (real) | float |
|---------------------|------------------|
| true | True |
|---------------------|------------------|
| false | False |
|---------------------|------------------|
| null | None |
|---------------------|------------------|
你不需要lambda
...而不是
value_deserializer=lambda m: json.loads(m)
你應該使用
value_deserializer=json.load
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.