Logstash File input: sincedb_path

Question

Upon restarting Logstash, at times observed that Logstash duplicates the log events. Was wondering as to what would be the right way to apply start_position,

Answer 1

start_position (beginning or end) is only used for files that have not yet been seen by logstash. The only reason to use 'beginning' is when you're trying to load older files.

sincedb_path just needs to be a directory where logstash has write permission for the registry.

sincedb_write_interval defines how often logstash should write the sincedb registry. A larger value puts you at risk in logstash were to crash.

When you have multiple files that match your glob, logstash tracks them separately by having multiple entries in the registry.

The registry contains the inode number, so logstash knows what to do in that type of rotation.