I am working on logging with FluentD and Graylog GELF with limited success. I want to forward a JSON file:
<source>
@type tail
path /var/log/suricata/eve.json
pos_file /var/log/td-agent/suri_eve.pos # pos record
tag ids
format json
# JSON time stamp: 2016-02-01T11:52:49.157072+0000
# this timestamp is ruby's t.strftime("%Y-%m-%dT%H:%M:%S.%6N%z")
time_format %Y-%m-%dT%H:%M:%S.%6N%z
time_key timestamp # I show a JSON message below
</source>
<match **>
@type graylog
host 1.2.3.4 #(optional; default="localhost")
port 12201 #(optional; default=9200)
flush_interval 30
num_threads 2
</match>
This kicks in, but produces error messages:
2016-02-01 15:30:11 +0000 [warn]: plugin/in_tail.rb:263:rescue in convert_line_to_event: "{\\"timestamp\\":\\"2016-02-01T15:27:09.000087+0000\\",\\"flow_id\\":51921072,\\"event_type\\":\\"flow\\",\\"src_ip\\":\\"10.1.1.85\\",\\"src_port\\":59820,\\"dest_ip\\":\\"224.0.0.252\\",\\"dest_port\\":5355,\\"proto\\":\\"UDP\\",\\"flow\\":{\\"pkts_toserver\\":4,\\"pkts_toclient\\":0,\\"bytes_toserver\\":294,\\"bytes_toclient\\":0,\\"start\\":\\"2016-02-01T15:26:30.393371+0000\\",\\"end\\":\\"2016-02-01T15:26:37.670904+0000\\",\\"age\\":7,\\"state\\":\\"new\\",\\"reason\\":\\"timeout\\"}}" error="invalid time format: value = 2016-02-01T15:27:09.000087+0000, error_class = ArgumentError, error = invalid strptime format - `%Y-%m-%dT%H:%M:%S.%6N%z'"
An original messages looks like this:
{"timestamp":"2016-02-01T15:31:02.000699+0000","flow_id":52015920,"event_type":"flow","src_ip":"10.1.1.44","src_port":49313,"dest_ip":"224.0.0.252","dest_port":5355,"proto":"UDP","flow":{"pkts_toserver":2,"pkts_toclient":0,"bytes_toserver":128,"bytes_toclient":0,"start":"2016-02-01T15:30:31.348568+0000","end":"2016-02-01T15:30:31.759024+0000","age":0,"state":"new","reason":"timeout"}}
So I checked the Ruby docs . I am not too familiar with FluentD but from what I know the time format expression should fit? I tried format=none
but that also doesn't work.
https://github.com/Graylog2/graylog2-server/issues/1761
This is a bug/problem with reserved fields (undocumented) in Graylog2. If you find a similar bug with timestamps, check the linked issue and the dev response.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.