从CSV文件输入数据到logstash

Question

我有一个csv文件，包含以下标题：

"PacketId","MACAddress","Date","PacketLength","SourceIP","SourcePort","DestIP","DestPort"

我想使用LogStash将数据索引到ElasticSearch，并且无法为其编写过滤器。

filter {
    grok {
        match => message => "%{IP:SourceIP}"
    }
    }

上面的过滤器提供了一个很好的SourceIP字段提取，但是我如何编写grok模式来为所有字段提取它。

Answer 1

让以下CSV文件：

1,00-14-22-01-23-45,13/09/2015,32,128.248.1.43,9980,128.248.23.13,9880
1,01-74-02-84-13-98,14/09/2015,64,128.248.1.94,9280,128.248.13.84,9380

您必须在此处设置Logstash配置：

input {
    file {
        path => "/path/of/your/csv/test.csv"
        sincedb_path => "/path/of/your/csv/test.idx"
        start_position => "beginning"
    }
}

filter {
    csv {
        separator => ","
        columns => ["PacketId","MACAddress","Date","PacketLength","SourceIP","SourcePort","DestIP","DestPort"]
    }
}

output {
    stdout {
        codec => rubydebug      
    }
}

您将获得输出结果：

{
         "message" => [
        [0] "1,00-14-22-01-23-45,13/09/2015,32,128.248.1.43,9980,128.248.23.13,9880"
    ],
        "@version" => "1",
      "@timestamp" => "2015-09-14T20:11:28.976Z",
            "host" => "MyHost.local",
            "path" => "/path/of/your/csv/test.csv",
        "PacketId" => "1",
      "MACAddress" => "00-14-22-01-23-45",
            "Date" => "13/09/2015",
    "PacketLength" => "32",
        "SourceIP" => "128.248.1.43",
      "SourcePort" => "9980",
          "DestIP" => "128.248.23.13",
        "DestPort" => "9880"
}
{
         "message" => [
        [0] "1,01-74-02-84-13-98,14/09/2015,64,128.248.1.94,9280,128.248.13.84,9380"
    ],
        "@version" => "1",
      "@timestamp" => "2015-09-14T20:11:28.978Z",
            "host" => "MyHost.local",
            "path" => "/path/of/your/csv/test.csv",
        "PacketId" => "1",
      "MACAddress" => "01-74-02-84-13-98",
            "Date" => "14/09/2015",
    "PacketLength" => "64",
        "SourceIP" => "128.248.1.94",
      "SourcePort" => "9280",
          "DestIP" => "128.248.13.84",
        "DestPort" => "9380"
}

此致，阿兰

Answer 2

您需要先使用CSV过滤器 ，而不是grok。