简体   繁体   English

如何使用有效时间格式将FluentD的JSON文件转发到Graylog2

[英]How to forward a JSON file with FluentD to Graylog2 with a valid time format

I am working on logging with FluentD and Graylog GELF with limited success. 我正在使用FluentD和Graylog GELF进行日志记录,但收效有限。 I want to forward a JSON file: 我想转发一个JSON文件:

<source>
  @type tail
  path /var/log/suricata/eve.json
  pos_file /var/log/td-agent/suri_eve.pos # pos record
  tag ids
  format json
  # JSON time stamp: 2016-02-01T11:52:49.157072+0000
  # this timestamp is ruby's t.strftime("%Y-%m-%dT%H:%M:%S.%6N%z")
  time_format %Y-%m-%dT%H:%M:%S.%6N%z
  time_key timestamp # I show a JSON message below
</source>

<match **>
  @type graylog
  host 1.2.3.4 #(optional; default="localhost")
  port 12201 #(optional; default=9200)
  flush_interval 30
  num_threads 2
</match>

This kicks in, but produces error messages: 这会启动,但会产生错误消息:

2016-02-01 15:30:11 +0000 [warn]: plugin/in_tail.rb:263:rescue in convert_line_to_event: "{\\"timestamp\\":\\"2016-02-01T15:27:09.000087+0000\\",\\"flow_id\\":51921072,\\"event_type\\":\\"flow\\",\\"src_ip\\":\\"10.1.1.85\\",\\"src_port\\":59820,\\"dest_ip\\":\\"224.0.0.252\\",\\"dest_port\\":5355,\\"proto\\":\\"UDP\\",\\"flow\\":{\\"pkts_toserver\\":4,\\"pkts_toclient\\":0,\\"bytes_toserver\\":294,\\"bytes_toclient\\":0,\\"start\\":\\"2016-02-01T15:26:30.393371+0000\\",\\"end\\":\\"2016-02-01T15:26:37.670904+0000\\",\\"age\\":7,\\"state\\":\\"new\\",\\"reason\\":\\"timeout\\"}}" error="invalid time format: value = 2016-02-01T15:27:09.000087+0000, error_class = ArgumentError, error = invalid strptime format - `%Y-%m-%dT%H:%M:%S.%6N%z'" 2016-02-01 15:30:11 +0000 [警告]:plugin / in_tail.rb:263:convert_line_to_event中的救援:“ {\\“ timestamp \\”:\\“ 2016-02-01T15:27:09.000087 + 0000 \\ ” \\ “flow_id \\”:51921072,\\ “EVENT_TYPE \\”:\\ “流\\”,\\ “src_ip \\”:\\ “10.1.1.85 \\”,\\ “src_port \\”:59820,\\ “DEST_IP \\”: \\ “224.0.0.252 \\”,\\ “dest_port \\”:5355,\\ “原\\”:\\ “UDP \\”,\\ “流\\”:{\\ “pkts_toserver \\”:4,\\ “pkts_toclient \\”:0 ,\\ “bytes_toserver \\”:294,\\ “bytes_toclient \\”:0,\\ “启动\\”:\\ “2016-02-01T15:26:30.393371 + 0000 \\”,\\ “端\\”:\\“2016-02 -01T15:26:37.670904 + 0000 \\“,\\”年龄\\“:7,\\”状态\\“:\\”新\\“,\\”原因\\“:\\”超时\\“}}”“ error =”无效的时间格式:值= 2016-02-01T15:27:09.000087 + 0000,error_class = ArgumentError,错误=无效的strptime格式-'%Y-%m-%dT%H:%M:%S.%6N%z'“

An original messages looks like this: 原始消息如下所示:

 {"timestamp":"2016-02-01T15:31:02.000699+0000","flow_id":52015920,"event_type":"flow","src_ip":"10.1.1.44","src_port":49313,"dest_ip":"224.0.0.252","dest_port":5355,"proto":"UDP","flow":{"pkts_toserver":2,"pkts_toclient":0,"bytes_toserver":128,"bytes_toclient":0,"start":"2016-02-01T15:30:31.348568+0000","end":"2016-02-01T15:30:31.759024+0000","age":0,"state":"new","reason":"timeout"}} 

So I checked the Ruby docs . 所以我检查了Ruby文档 I am not too familiar with FluentD but from what I know the time format expression should fit? 我对FluentD不太熟悉,但是据我所知,时间格式表达式应该适合吗? I tried format=none but that also doesn't work. 我尝试过format=none但这也行不通。

https://github.com/Graylog2/graylog2-server/issues/1761 https://github.com/Graylog2/graylog2-server/issues/1761

This is a bug/problem with reserved fields (undocumented) in Graylog2. 这是Graylog2中具有保留字段(未记录)的错误/问题。 If you find a similar bug with timestamps, check the linked issue and the dev response. 如果您发现带有时间戳的类似错误,请检查链接的问题和开发人员的响应。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM