Logstash reading file from begining?

Question

I am facing issue with Logstash file input.

Config initially:

input {
     file{
                path => "/home/Desktop/Logstash-Input/**/*_log"
                start_position => "beginning"
                sincedb_path => "/dev/null"
        }
}

I do this ( sincedb_path => "/dev/null" ) so that logstash forgets previous file offsets. So logstash reads all the logs files.

Now I restart logstash with sincedb_path => "/dev/null" removed, so now logstash starts with no new files read (as per my logstash console).

Now, when I append a log line to a file, it should read only the last appended line, but it starts reading the file from somewhere in between.

Logstash console :

/home/Desktop/Logstash-Input/Server2/CronLog/2014.05.30/cron_log: **old inode** was [**2754731**, 0, 2054], new is [**2754740**, 0, 2054] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"73"}

:delete for /home/Desktop/Logstash-Input/Server2/CronLog/2014.05.30/cron_log, deleted from @files {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"85"}

**writing sincedb** (delta since last write = 190) {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"177"}
_open_file: /home/Desktop/Logstash-Input/Server2/CronLog/2014.05.30/cron_log: opening {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"98"}

/home/Desktop/Logstash-Input/Server2/CronLog/2014.05.30/cron_log: sincedb last value **741**, cur size 2488 {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"122"}

/home/Desktop/Logstash-Input/Server2/CronLog/2014.05.30/cron_log: sincedb: seeking to 741 {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"124"}

Received line {:path=>"/home/Desktop/Logstash-Input/Server2/CronLog/2014.05.30/cron_log", :text=>"5-26T00:00:01+05:30 bx920as1 crond[26393]: (user6) CMD (sh 2014/05/30/server2/cron/log)", :level=>:debug, :file=>"logstash/inputs/file.rb", :line=>"134"}

filter received {:event=>{"message"=>"5-26T00:00:01+05:30 bx920as1 crond[26393]: (user6) CMD (sh 2014/05/30/server2/cron/log)", "@version"=>"1", "@timestamp"=>"2014-10-31T07:00:44.404Z", "host"=>"cdot-HP-Pro-3330-MT", "path"=>"/home/Desktop/Logstash-Input/Server2/CronLog/2014.05.30/cron_log"}, :level=>:debug, :file=>"(eval)", :line=>"33"}

My sincedb file contents before adding sincedb_path => "/dev/null" was:

2754732 0 2054 1178
2755530 0 2054 249
2754730 0 2054 495
2755274 0 2054 548
2624449 0 2054 1140
2755536 0 2054 248
**2754731 0 2054 2396**
2754735 0 2054 1358
2755539 0 2054 239
2755542 0 2054 316
2755540 0 2054 316
2887689 0 2054 3482
2754736 0 2054 584
2754737 0 2054 675
2754738 0 2054 438
2754739 0 2054 546
2754734 0 2054 1269
2754740 0 2054 577
**2754741 0 2054 2304**
2754733 0 2054 659

and sincedb file after removing sincedb_path => "/dev/null" :

2754732 0 2054 1178
2755530 0 2054 249
2754730 0 2054 905
2755274 0 2054 548
2624449 0 2054 1140
2755536 0 2054 248
2754731 0 2054 2396
2754735 0 2054 1358
2755539 0 2054 239
2755542 0 2054 316
2755540 0 2054 316
2887689 0 2054 3482
2754736 0 2054 584
2754737 0 2054 675
2754738 0 2054 438
2754739 0 2054 546
2754734 0 2054 1269
**2754740 0 2054 741
2754741 0 2054 2304**
2754733 0 2054 823

Why is logstash reading the file from in between? Plz help.

Answer 1

As stated in the doc , start_position is only used for new files. Logstash determines whether a file is new by checking the sincdb file for the inode.

From your log file fragment, logstash sees cron_log is inode 2754740. Now, note that this node exists in the first (pre-restart) sincedb file:

2754740 0 2054 577

When you restart logstash with a real (or default) sincedb location, it will locate the inode in the file and start at byte offset 577.

If you're just coming out of testing, make sure your inodes are not listed in the sincedb file. In my environment, I remove the sincedb file when I need to, though that might not work in your environment.