Flume HTTPSource由于格式错误而拒绝JSON正文

Question

我有一个Flume代理配置，在其中使用HTTPSource接收来自特定服务的事件数据。 为了进行测试，我在Python中将静态JSON结构创建为一个名为data的字符串对象（请参见下面的代码片段1），然后将该对象发送到带有适当标头的flume中，但是flume却向我返回了400错误的请求错误时间（请参见下面的代码段2）。 下面的代码段3中提供了相关的水槽执行异常消息。

问题：谁能告诉我导致flume HTTPSource拒绝的静态json请求出了什么问题？ 我可能还缺少其他一些与json数据无关的问题吗？ 谢谢。

SNIPPET 1（Python脚本生成包含我的静态JSON数据的虚拟HTTP请求）

 import urllib2, json

 serviceName = "serviceA"
 timestamp = datetime.datetime.strftime(datetime.datetime.now(), '%Y-%m-%d %H:%M:%S')

 body = "{ \"service\":\"" + serviceName + "\" }"
 print("BODY: " +  body)

 //My JSON data
 data = "[{ \"headers\" : { \"timestamp\" : \"" + timestamp + "\" }, \"body\" : " + body + " }]"
 print("DATA: "  + data)

 req = urllib2.Request("http://10.1.0.100:5140")
 req.add_header('Content-Type', 'application/json')
 response = urllib2.urlopen(req, data)  

SNIPPET 2-python脚本的执行输出

  BODY: { "service":"serviceA" }
  DATA: [{ "headers" : { "timestamp" : "2016-01-13 12:26:48" }, "body" : { "service":"serviceA" } }]
  Traceback (most recent call last):
  File "./event_data_gen.py", line 57, in <module>
  response = urllib2.urlopen(req, data)  
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
  return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
  response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
  'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
  return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
  result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
  raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
  urllib2.HTTPError: HTTP Error 400: Bad request from client. 
  Request has invalid JSON Syntax.

SNIPPET 3-Flume执行输出中的异常消息

  2016-01-13 12:26:48,653 (34313572@qtp-604003190-13)
  [WARN -     org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:      242)] Received bad request from client. 
  org.apache.flume.source.http.HTTPBadRequestException: Request has invalid JSON Syntax.
  at org.apache.flume.source.http.JSONHandler.getEvents(JSONHandler.java:119)
  at  org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:240)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
  at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
  at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
  at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
  at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
  at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
  at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
  at org.mortbay.jetty.Server.handle(Server.java:326)
  at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
  at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
  at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
  at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
  at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
  at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
  at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
  Caused by: com.google.gson.JsonSyntaxException:  java.lang.IllegalStateException: Expected a string but was BEGIN_OBJECT at line  1 column 67
  at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176)
  at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
  at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:81)
  at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:60)
  at com.google.gson.Gson.fromJson(Gson.java:795)
  at com.google.gson.Gson.fromJson(Gson.java:761)
  at org.apache.flume.source.http.JSONHandler.getEvents(JSONHandler.java:117)
... 17 more
  Caused by: java.lang.IllegalStateException: Expected a string but was  BEGIN_OBJECT at line 1 column 67
  at com.google.gson.stream.JsonReader.nextString(JsonReader.java:464)
  at com.google.gson.internal.bind.TypeAdapters$13.read(TypeAdapters.java:349)
  at com.google.gson.internal.bind.TypeAdapters$13.read(TypeAdapters.java:337)
  at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
  at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)

Answer 1

正如@Martyn W在其评论中指出的那样，JSONHandler期望并验证JSON具有某种结构。

JSONHandler的JavaDoc中对此进行了描述：

用于HTTPSource的JSONHandler，它接受事件数组。 如果反序列化由于格式错误或任何其他原因而失败，则此处理程序将引发异常。 每个事件都必须编码为具有两个键值对的映射。

1. 标头-此键值对的键是“标头”。 该键的值是另一个映射，表示事件标头。 这些标头按原样插入Flume事件。

2. 正文-正文是代表事件正文的字符串。 此键值对的键是“ body”。 所有键值对均视为标头。 一个例子：

[{“ headers”：{“ a”：“ b”，“ c”：“ d”}，“ body”：“ random_body”}，{“ headers”：{“ e”：“ f”}，“ body “：” random_body2“}]