grok filter and logstash

Question

There is a log of 2 types: 1.sshd: d.kuprevich@pts/0/79512/1012 124.25 111.561 2.74.205.155.180:52472-76.121.32.65:443/0/0 1.4234 11.4315 In the first case after sshd: space

Here is the config from https://grokdebug.herokuapp.com/ for the first and second case

1. (? <Ip, port>% {IPORHOST}% {NOTSPACE} \ s +% {NOTSPACE}) \ s +% {NOTSPACE: received_traffic} \ s +% {NOTSPACE: transmitted_traffic}
2. (? <Ip, port>% {IPORHOST}% {NOTSPACE}% {NOTSPACE}) \ s +% {NOTSPACE: received_traffic} \ s +% {NOTSPACE: transmitted_traffic} They are working each for their own case, but I need a universal one for both cases, but I can't think of something yet. Help.

And the second question, if possible, in the logstash config I have an input file, there are conditionally 5 lines in it, after the output to the kibana console I see all the same data 2 times, but with different IDs, please tell me how to get rid of duplication given. Thanks for the help.

Answer 1

Putting each number in a distinct field:

%{HOSTPORT:IP_1}-%{HOSTPORT:IP_2}/%{NUMBER:NUMBER_1}/%{NUMBER:NUMBER_2}%{SPACE}%{NUMBER:NUMBER_3}%{SPACE}%{NUMBER:NUMBER_4}

For your given log line, you'll get:

NUMBER_3    10.5678
NUMBER_2    0
NUMBER_1    0
NUMBER_4    1516.17
IP_2    14.212.55.167:80
IP_1    91.231.121.222:35712 

To help you next time you're trying to write a pattern:

pattern tester: http://grokconstructor.appspot.com/do/match

base pattern list: https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/ecs-v1/grok-patterns