I have a piece of code that monitors a directory for addition of files. Whenever a new file is added to the directory, the contents of the file are picked and published on kafka
I had similar problem as well as in the following thread: Multithreading on Queue when trying to upload a dynamically created files to a Queue service and it was taking 2 days for me to resolve. Thanks to Holger who gave answer as above whereby the locking occurs might be due to the creation had not fully done upon reading by another thread, it has saved me a lot of time.
My initial solution as found a lot from the internet, was:
WatchEvent ev = cast(event);
Path name = ev.context();
Path child = dir.resolve(name);
//queueUploadFile(child);
if (kind == ENTRY_CREATE) {
uploadToQueue(this.queueId, child);
}
I changed it to:
WatchEvent ev = cast(event);
Path name = ev.context();
Path child = dir.resolve(name);
//queueUploadFile(child);
if (kind == ENTRY_MODIFY) {
uploadToQueue(this.queueId, child);
}
And everything works perfectly. To handle the "somehow" multiple ENTRY_MODIFY events firing (duplicated files uploaded), I perform deletion on the file inside the uploadToQueue() method once it's uploaded.
I hope my approach taken based on the above contribution will also help others with similar problem.