堆棧內存溢出
  簡體   English   中英

Logstash 不保存其他字段

[英]Logstash not saving additional fields

 原文  2020-07-22 17:03:23  5   1    elasticsearch/ logstash/ logstash-grok/ elk

問題描述

在工作彈性堆棧上，我正在嘗試將 mysql 慢日志發送到 OSS elasticsearch 集群。 數據保存到 ES，例如包括[event][dataset] ，但是[mysql][slowlog]甚至[mysql] arrays 下的所有內容都被丟棄。 這是我的管道配置：

input {
    beats {
        port => 5404
        ssl => true
        client_inactivity_timeout => 240
        ssl_certificate_authorities => ["........pem"]
        ssl_certificate => ".........pem"
        ssl_key => "...........pem"
        ssl_verify_mode => "force_peer"
    }
}
filter {
  if [fileset][module] == "mysql" {
    if [fileset][name] == "error" {
      # [mysql][error] part, not that relevant
    }
    else if [fileset][name] == "slowlog" {
      grok {
        match => { "message" => ["^# User@Host: %{USER:[mysql][slowlog][user]}(\[[^\]]+\])? @\s* \[%{IP:[mysql][slowlog][host]}\] \s*Id:\s* %{NUMBER:[mysql][slowlog][id]}\n# Schema: %{GREEDYDATA:[mysql][slowlog][schema]}\s* Last_errno: %{NUMBER:[mysql][slowlog][last_errno]}\s* Killed: %{NUMBER:[mysql][slowlog][killed]}\n# Query_time: %{NUMBER:[mysql][slowlog][query_time][sec]}\s* Lock_time: %{NUMBER:[mysql][slowlog][lock_time][sec]}\s* Rows_sent: %{NUMBER:[mysql][slowlog][rows_sent]}\s* Rows_examined: %{NUMBER:[mysql][slowlog][rows_examined]}\s* Rows_affected: %{NUMBER:[mysql][slowlog][rows_affected]}\n# Bytes_sent: %{NUMBER:[mysql][slowlog][bytes_sent]}\n(SET timestamp=%{NUMBER:[mysql][slowlog][timestamp]};\n)?%{GREEDYMULTILINE:[mysql][slowlog][query]}"] }
        pattern_definitions => {
          "GREEDYMULTILINE" => "(.|\n)*"
        }
        remove_field => "message"
      }
      date {
        match => [ "[mysql][slowlog][timestamp]", "UNIX" ]
      }
      mutate {
        gsub => ["[mysql][slowlog][query]", "\n# Time: [0-9]+ [0-9][0-9]:[0-9][0-9]:[0-9][0-9](\\.[0-9]+)?$", ""]
        add_field => {"locktime" => "%{[mysql][slowlog][lock_time][sec]}"}
      }
    }
  }
}
output {
    elasticsearch {
      user => "logstash_writer"
      password => "******"
      hosts => ["localhost:9200", "localhost:9201"]
      index => "%{[@metadata][beat]}-%{+YYYY.MM}"
  }
}

即使我嘗試將一些值保存到duration字段而不是[mysql][slowlog][query_time][sec] ，或者將[mysql][slowlog][lock_time][sec]的值復制到新字段locktime ，字段不會'不要在 ES 上創建。 幾天來，我真的一直在准備文檔並瀏覽類似的問題，但不知道我的管道出了什么問題……這是在 ES 上創建的映射：

{
  "mapping": {
    "_doc": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "agent": {
          "properties": {
            "ephemeral_id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "hostname": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "event": {
          "properties": {
            "dataset": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "module": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "fileset": {
          "properties": {
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "host": {
          "properties": {
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "input": {
          "properties": {
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "flags": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "offset": {
              "type": "long"
            }
          }
        },
        "message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "service": {
          "properties": {
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "tags": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

這是 ES 條目的示例：

{
  "took" : 346,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1220,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-2020.07",
        "_type" : "_doc",
        "_id" : "SSYId3MBlOlESbQxaGQs",
        "_score" : 1.0,
        "_source" : {
          "host" : {
            "name" : "nos12"
          },
          "input" : {
            "type" : "log"
          },
          "agent" : {
            "version" : "7.6.1",
            "type" : "filebeat",
            "ephemeral_id" : "1decb58b-dc7e-4bf9-8942-df203ef1504f",
            "hostname" : "nos12",
            "id" : "ea3f44c8-faa3-48e5-869c-3a4703c67818"
          },
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "event" : {
            "dataset" : "mysql.slowlog",
            "module" : "mysql"
          },
          "@timestamp" : "2020-07-22T14:47:44.559Z",
          "fileset" : {
            "name" : "slowlog"
          },
          "message" : """# User@Host: pmm[pmm] @  [127.0.0.1]  Id: 5737606019
# Schema:   Last_errno: 0  Killed: 0
# Query_time: 0.000215  Lock_time: 0.000052  Rows_sent: 0  Rows_examined: 0  Rows_affected: 0
# Bytes_sent: 361
SET timestamp=1595429264;
SELECT /* pmm-agent:perfschema */ `performance_schema`.`events_statements_history`.`SQL_TEXT`, `performance_schema`.`events_statements_history`.`DIGEST`, `performance_schema`.`events_statements_history`.`CURRENT_SCHEMA` FROM `performance_schema`.`events_statements_history` WHERE DIGEST IS NOT NULL AND SQL_TEXT IS NOT NULL;""",
          "service" : {
            "type" : "mysql"
          },
          "log" : {
            "flags" : [
              "multiline"
            ],
            "file" : {
              "path" : "/mysql_data/slowquery/slow.log"
            },
            "offset" : 10146495
          },
          "@version" : "1",
          "ecs" : {
            "version" : "1.4.0"
          }
        }
      }
    ]
  }
}

慢日志示例：

# Schema:   Last_errno: 0  Killed: 0
# Query_time: 0.000211  Lock_time: 0.000055  Rows_sent: 0  Rows_examined: 0  Rows_affected: 0
# Bytes_sent: 361
SET timestamp=1595435719;
SELECT /* pmm-agent:perfschema */ `performance_schema`.`events_statements_history`.`SQL_TEXT`, `performance_schema`.`events_statements_history`.`DIGEST`, `performance_schema`.`events_statements_history`.`CURRENT_SCHEMA` FROM `performance_schema`.`events_statements_history` WHERE DIGEST IS NOT NULL AND SQL_TEXT IS NOT NULL;

grok 調試器結果：

{
  "[mysql][slowlog][rows_affected]": "0",
  "[mysql][slowlog][schema]": " ",
  "[mysql][slowlog][killed]": "0",
  "[mysql][slowlog][query]": "SELECT /* pmm-agent:perfschema */ `performance_schema`.`events_statements_history`.`SQL_TEXT`, `performance_schema`.`events_statements_history`.`DIGEST`, `performance_schema`.`events_statements_history`.`CURRENT_SCHEMA` FROM `performance_schema`.`events_statements_history` WHERE DIGEST IS NOT NULL AND SQL_TEXT IS NOT NULL;",
  "[mysql][slowlog][rows_examined]": "0",
  "[mysql][slowlog][host]": "127.0.0.1",
  "[mysql][slowlog][rows_sent]": "0",
  "[mysql][slowlog][timestamp]": "1595429264",
  "[mysql][slowlog][id]": "5737606019",
  "[mysql][slowlog][last_errno]": "0",
  "[mysql][slowlog][lock_time][sec]": "0.000052",
  "[mysql][slowlog][user]": "pmm",
  "[mysql][slowlog][query_time][sec]": "0.000215",
  "[mysql][slowlog][bytes_sent]": "361"
}

有趣的是，即使使用無效的 grok 模式，我也會得到相同的結果，並且沒有其他管道從 mysql 捕獲文件節拍數據：/

1 個解決方案

解決方案1
 0 已采納  2020-07-23 15:19:18

來自討論jenni的 jenni 找到了答案 - 解決方案是將[fileset][module] (不存在) 更改為[event][module] ：

https://discuss.elastic.co/t/logstash-not-saving-additional-fields/242220[1]

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 Logstash輸入和輸出字段 Logstash不添加字段 如何將額外的 jars 傳遞給 LogStash 類路徑 如何不通過logstash解析某些字段？ Logstash：映射中的字段順序錯誤 使用Logstash自動解析日志字段 Logstash無法添加字段？ Logstash .raw字段未出現在Elasticsearch中 Logstash中的節拍輸入丟失了字段 如何利用Logstash索引數據，但不從Logstash生成額外字段
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM