Filebeat - 仅在第一次输入日志文件时缺少打开“{”？

Question

I have a JSON log file that looks like this:

{"timestamp": "2021-03-20 13:56:07,568", "action": "ws.RegisterUser", "uuid": "e46bfe57-083d-43f9-86fa-4c94626032fe", "application_id": "App_Caller", "ip": "1.2.3.4", "log_level": "INFO", "event": "request", "system": "App_Host", "user_id": "1234567"}
{"timestamp": "2021-03-20 13:56:07,679", "action": "ws.RegisterUser", "uuid": "e46bfe57-083d-43f9-86fa-4c94626032fe", "application_id": "App_Caller", "ip": "1.2.3.4", "log_level": "INFO", "event": "request", "system": "Integration_1", "integration_action": "Create User", "user_id": "1234567"}
{"timestamp": "2021-03-20 13:56:07,789", "action": "ws.RegisterUser", "uuid": "e46bfe57-083d-43f9-86fa-4c94626032fe", "application_id": "App_Caller", "ip": "1.2.3.4", "log_level": "INFO", "event": "response", "system": "Integration_1", "integration_action": "Create User", "result": "01", "user_id": "1234567"}

Running this through Filebeat (v7.12.0) on Ubuntu 20.10, I keep getting this error:

"Error decoding JSON: invalid character 'i' in literal true (expecting 'r')"

always on the first JSON item, the other two are decoded successfully.

I've changed the order of the items but the error always occurs on whichever happens to be the first.

I then commented the JSON configs, and noticed that the Filebeat output is missing the opening "{" on the first line:

...
  "message": "timestamp\": \"2021-03-20 13:56:07,568\", \"action\": \"ws.RegisterUser\", \"uuid\": \"e46bfe57-083d-43f9-86fa-4c94626032fe\", \"application_id\": \"App_Caller\", \"ip\": \"1.2.3.4\", \"log_level\": \"INFO\", \"event\": \"request\", \"system\": \"App_Host\", \"user_id\": \"1234567\"}",
...
  "message": "{\"timestamp\": \"2021-03-20 13:56:07,679\", \"action\": \"ws.RegisterUser\", \"uuid\": \"e46bfe57-083d-43f9-86fa-4c94626032fe\", \"application_id\": \"App_Caller\", \"ip\": \"1.2.3.4\", \"log_level\": \"INFO\", \"event\": \"request\", \"system\": \"Integration_1\", \"integration_action\": \"Create User\", \"user_id\": \"1234567\"}",
...
  "message": "{\"timestamp\": \"2021-03-20 13:56:07,789\", \"action\": \"ws.RegisterUser\", \"uuid\": \"e46bfe57-083d-43f9-86fa-4c94626032fe\", \"application_id\": \"App_Caller\", \"ip\": \"1.2.3.4\", \"log_level\": \"INFO\", \"event\": \"response\", \"system\": \"Integration_1\", \"integration_action\": \"Create User\", \"result\": \"01\", \"user_id\": \"1234567\"}",
...

This is my config file:

filebeat:
  config:
    modules:
      path: /etc/filebeat/modules.d/*.yml
      reload:
        enabled: false
  inputs:
  - enabled: false
    paths:
    - /var/log/*.log
    type: log
  - enabled: false
    paths:
    - /var/log/*.log
    type: filestream
  - enabled: true
    paths:
    - /home/kubs/Documents/fb/test_*.log
#    json.message_key: event
#    json.add_error_key: true
    type: log
output:
#  elasticsearch:
#    hosts:
#    - localhost:9200
  console:
    pretty: true
path:
  config: /etc/filebeat
  data: /var/lib/filebeat
  home: /usr/share/filebeat
  logs: /var/log/filebeat
processors:
- add_host_metadata:
    when:
      not:
        contains:
          tags: forwarded
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup:
  kibana: null
  template:
    settings:
      index:
        number_of_shards: 1

I've done some searches, and found similar cases of the JSON decoding error message (but never in a scenario as simple as mine), but found no mention of any missing "{" on first line.

I suspect the issues may be related, but I'm not really sure.

Am I missing something obvious that could explain this behaviour, and the cause of the JSON decoding error?

Answer 1

As is usually the case, a few more minutes of re-reading my question could've helped me come up with something else to try.

I've had some issues with Filebeat registry files (Logstash was so much simpler), so I've come up with a convoluted way to repeatedly process the same file, involving "clearing" it with an echo 1 > $file , deleting it and restoring it to its former contents with a new name, along with stopping/starting Filebeat.

So, I've just deleted all the log files, created a new file with the same contents, changed my config from this

paths:
- /home/kubs/Documents/fb/test_*.log

to this (specify the actual file)

paths:
- /home/kubs/Documents/fb/test_20210331153923.log

and now everything works as expected, including the JSON decoding.

I don't know yet what's causing the missing "{", so if anyone has any ideas, I'd like to hear them. Otherwise, I'll just accept this as an answer in a couple of days.

Edit: It definitely has to do with the way I was running my tests, the way Filebeat keeps state and the reuse of inodes on Linux.

Here's the last three entries on the registry file:

{"op":"set","id":150}
{"k":"filebeat::logs::native::406026-2051","v":{"id":"native::406026-2051","prev_id":"","source":"/home/kubs/Documents/fb/test_20210331162059.log","offset":2,"timestamp":[258565806201,1617204774],"ttl":-1,"type":"log","FileStateOS":{"inode":406026,"device":2051},"identifier_name":"native"}}
{"op":"set","id":151}
{"k":"filebeat::logs::native::406026-2051","v":{"type":"log","FileStateOS":{"inode":406026,"device":2051},"identifier_name":"native","prev_id":"","source":"/home/kubs/Documents/fb/test_20210331163246.log","timestamp":[258567741346,1617204774],"ttl":-1,"id":"native::406026-2051","offset":2}}
{"op":"set","id":152}
{"k":"filebeat::logs::native::406026-2051","v":{"timestamp":[258575697063,1617204775],"FileStateOS":{"inode":406026,"device":2051},"identifier_name":"native","prev_id":"","source":"/home/kubs/Documents/fb/test_20210331163246.log","ttl":-1,"type":"log","id":"native::406026-2051","offset":858}}

The first entry (id: 150) has source "/home/kubs/Documents/fb/test_20210331162059.log" and offset 2 (which is after I "clear" the file). The "FileStateOS" is {"inode":406026,"device":2051}.

After I delete the file and create the new one, we get the next entry (id: 151), which has source "/home/kubs/Documents/fb/test_20210331163246.log" (the new log file, with the JSON contents), but still has the same "FileStateOS":{"inode":406026,"device":2051}, ie, the inode was reused, and has the same offset.

And that's why the "{" was always missing on the first line, because the offset was always 2.