堆栈内存溢出
  简体   繁体   English

Logstash-解析和变异JSON文件

[英]Logstash - Parsing and mutating JSON file

 原文  2016-10-20 14:04:28  2   1    json/ elasticsearch/ logstash

问题描述

Having the following JSON file: 具有以下JSON文件： 

{
  "count": 2,
  "status": {"partial": true},
  "records": [
    {
      "info": {
        "startTime": "2016-07-17 08:42:40.212+0000",
        "endTime": "2016-07-17 08:43:47.715+0000",
        "id": "123456789"
      },
      "conversation": {
        "lines": [
          {
            "time": "2016-07-17 08:42:32.533+0000",
            "text": "Hi There",
            "user": "user A"
          },
          {
            "time": "2016-07-17 08:42:36.533+0000",
            "text": "Hello",
            "user": "user B"
          }
        ]
      }
    },
    {
      "info": {
        "startTime": "2016-07-18 08:42:40.212+0000",
        "endTime": "2016-07-18 08:43:47.715+0000",
        "id": "4567890"
      },
      "conversation": {
        "lines": [
          {
            "time": "2016-07-17 08:42:32.533+0000",
            "text": "Hi There",
            "user": "user X"
          },
          {
            "time": "2016-07-17 08:42:36.533+0000",
            "text": "Hello",
            "user": "user Y"
          }
        ]
      }
    }
  ]
}

EDIT (raw format): 编辑（原始格式）： 

{"count": 20,"status": {"partial": true},"records": [{"info": {"startTime": "2016-07-17 08:42:40.212+0000","endTime": "2016-07-17 08:43:47.715+0000","id": "123456789"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user A"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user B"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}}
]}

I would like to use logstash to import conversation.lines (ignoring the rest of the information like info ) for each of the records, and maybe run some logic like removing some of the lines depending on the time property. 我想用logstash导入conversation.lines （忽略像其余的信息info ）每次的记录，也许运行一些逻辑，如消除一些因时间属性线。

Is it possible to do that with Logstash alone, or should I preprocess the file? 是否可以仅使用Logstash来执行此操作，还是应该预处理文件？

1 个解决方案

解决方案1
 2 已采纳  2016-10-20 15:02:32

I think the easiest would be to use node.js. 我认为最简单的方法是使用node.js。

  1. require your JSON file require您的JSON文件
  2. loop over the records 循环records
  3. loop over the conversation.lines 循环conversation.lines
  4. apply your logic 运用你的逻辑
  5. send each line to ES with the JS ES client 使用JS ES客户端将每一行发送到ES

Logstash is very good at parsing text files line by line, but if you want to parse a multi-line JSON file, that would not be my first choice. Logstash非常擅长逐行解析文本文件，但是如果您想解析多行JSON文件，那不是我的首选。

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 将JSON文件解析为logstash - Parsing JSON file into logstash 在Logstash中解析JSON事件 - Parsing JSON event in Logstash Logstash麻烦解析Json - Logstash Trouble parsing Json Logstash不解析JSON - Logstash not parsing JSON JSON数组解析问题与logstash - json array parsing issue with logstash 从 RabbitMQ 解析 Logstash JSON - Logstash JSON Parsing from RabbitMQ 在 Logstash 中解析出尴尬的 JSON - Parsing out awkward JSON in Logstash logstash json过滤器未解析获取_jsonparsefailure的字段 - logstash json filter not parsing fields getting _jsonparsefailure Logstash JSON解析后删除字段 - Logstash remove field after JSON-parsing it 从 Logstash 中的日志文件解析一组 JSON 对象 - Parsing an array of JSON objects from logfile in Logstash
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM