Calculate Line Number in Log with Logstash Pipeline

Question

Imagine that have some logs like this

Jun2012|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|
Jun2012|17:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|
Jun2013|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|
Jun2012|18:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|
Jun2012|19:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|

..... I can see the line number in an editor but I want to calculate line number of each line and make field of it.In out put inside a json like

{...,"line_num" : 23, ...} 

How can i make this? Can someone give an example?

Answer 1

Ruby has a special variable $. that has the line number.

File.foreach(logfile).each do |line|
   puts "#{$.}: #{line}"
end

Answer 2

Try the below with readlines and with_index

result = []
File.readlines('test.log').each.with_index(1) do |line, index|
  result << { content: line, line_num: index }
end

Output:

2.6.3 :042 > pp result
[{:content=>
   "Jun2012|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|\n",
  :line_num=>1},
 {:content=>
   "Jun2012|17:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|\n",
  :line_num=>2},
 {:content=>
   "Jun2013|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|\n",
  :line_num=>3},
 {:content=>
   "Jun2012|18:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|\n",
  :line_num=>4},
 {:content=>
   "Jun2012|19:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|",
  :line_num=>5}]

Answer 3

Logstash has no information about the line number of the event in a file, and it only tracks the byte offset when you use the file input.

If you are using filebeat to ship your logs to logstash you also have the information about the byte offset, but there is no tracking of the actual line number.

If you want to just order your events you can use this offset, if you need the actual line number you will need to implement a way to add this information in your event before sending it to logstash.

Answer 4

I have solved that issue, with creating your own input codec plugin you can calculate line numbers via iterating over events