Logstash - Failed to open <file_path> Permission denied

Question

I am using logstash to push all the text logs from storage to elastic search. My storage size is about 1 TB. To Start with I have started to push 368 GB data (may be few hundred thousand files) to elastic search but logstash is failing with following error.

{:timestamp=>"2014-05-15T00:41:12.436000-0700", :message=>"/root/share/archive_data/sessionLogs/965c6f46-1a5e-4820-a68d-7c32886972fc/Log.txt: file grew, old size 0, new size 1557420", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"81"}
{:timestamp=>"2014-05-15T00:41:12.437000-0700", :message=>":modify for /root/share/archive_data/sessionLogs/965c6f46-1a5e-4820-a68d-7c32886972fc/Log.txt, does not exist in @files", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"77"}
{:timestamp=>"2014-05-15T00:41:12.441000-0700", :message=>"_open_file: /root/share/archive_data/sessionLogs/965c6f46-1a5e-4820-a68d-7c32886972fc/Log.txt: opening", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"98"}
{:timestamp=>"2014-05-15T00:41:12.441000-0700", :message=>"(warn supressed) failed to open /root/share/archive_data/sessionLogs/965c6f46-1a5e-4820-a68d-7c32886972fc/Log.txt: Permission denied - /root/share/archive_data/sessionLogs/965c6f46-1a5e-4820-a68d-7c32886972fc/Log.txt", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"110"}

share is network mounted. I am using root user to start logstash. User should have all the access needed to mount. share directory has following access drwxr-xr-x 44 root root 0 May 13 08:36 share

Now, my log files are static they don't change.

So, my question is - Is there anyway to let logstash know that do not store file handles once it process one log file. I think above error is occurred because number of log files is huge.

I have already filed a bug and there is existing bug in logstash which says that logstash doesn't do well when log files are more in number.

I see some duplicate issues here but I would like to know if anybody has any experience with this kind of issue?

Answer 1

I think, for logstash 1.4.2, the only answer is to:

• move or delete the files from the monitored directory
• restart logstash

I don't think there's any other way to have logstash release file handles of logs that have been processed and won't be added to any more.