Directory Scanner in Java

问题

Scan a set of directories continuously for a set of file name filters. For each file name filter arrived, process the file and repeat the steps for all

What can be the recommended design for this in jdk 1.5 , possibly using java.concurrent.Executor and Future

回答1:

I have done a similar task with the web crawler.Just a few changes had to be made... It is a concurrent implementation with newly found directories getting scanned by the thread pool in the Executor Framework.It uses concurrent collections for Queue and List to hold the indexed files. The indexer picks up the files from the queue and does something with them. here is the FileFilter implementation

    public class ImageFileFilter implements FileFilter
    {
      private final String[] okFileExtensions = 
        new String[] {"jpg", "png", "gif"};

      public boolean accept(File file)
      {
        for (String extension : okFileExtensions)
        {
          if (file.getName().toLowerCase().endsWith(extension))
          {
            return true;
          }
        }
        return false;
      }
    }

here is the Class with the main method...

     public class FileFilterTest {
        public static void main(String[] args) {
            File dir = new File("D:\\dev\\css-templates\\cms-admin");
            BlockingQueue blockingQueue = new ArrayBlockingQueue(5);
            FileCrawler fileCrawler = new FileCrawler(blockingQueue,
                    new ImageFileFilter(), dir);
            new Thread(fileCrawler).start();

            FileIndexer indexer = new FileIndexer(blockingQueue);
            new Thread(indexer).start();
        }
    }

Here is the file crawler thread

     public class FileCrawler implements Runnable {
            private final BlockingQueue fileQueue;
            private ConcurrentSkipListSet indexedFiles = new ConcurrentSkipListSet();
            private final FileFilter fileFilter;
            private final File root;
            private final ExecutorService exec = Executors.newCachedThreadPool();

            public FileCrawler(BlockingQueue fileQueue,
                               final FileFilter fileFilter,
                               File root) {
                this.fileQueue = fileQueue;
                this.root = root;
                this.fileFilter = new FileFilter() {
                    public boolean accept(File f) {
                        return f.isDirectory() || fileFilter.accept(f);
                    }
                };
            }

            public void run() {

                    submitCrawlTask(root);

            }

            private void submitCrawlTask(File f) {
                CrawlTask crawlTask = new CrawlTask(f);
                exec.execute(crawlTask);
            }

            private class CrawlTask implements Runnable {
                private final File file;

                CrawlTask(File file ) {

                    this.file= file;
                }

             public void run() {        
                 if(Thread.currentThread().isInterrupted())
                return;

                    File[] entries = file.listFiles(fileFilter);

                    if (entries != null) {
                        for (File entry : entries)
                            if (entry.isDirectory())
                                submitCrawlTask(entry);
                            else if (entry !=null && !indexedFiles.contains(entry)){
                                indexedFiles.add(entry);
                                try {
                                    fileQueue.put(entry);
                                } catch (InterruptedException e) {
                                        Thread.currentThread().interrupt();
                                }
                            }
                    }
                }
        }
       }

Here is the file indexer thread

        public class FileIndexer implements Runnable {
        private final BlockingQueue queue;

        public FileIndexer(BlockingQueue queue) {
            this.queue = queue;
        }

        public void run() { 
            try {
                while (true) {
                    indexFile(queue.take());
                }
            } catch (InterruptedException e) {
                System.out.println("Indexer Interrupted");
                Thread.currentThread().interrupt();
            }
        }

        public void indexFile(File file) {
            // do something with the file...
            System.out.println("Indexing File : " + file.getAbsolutePath() + " " + file.getName());
        };
    }

回答2:

I guess this is what you are trying to do:

1. You have a set of dirs:

   dir1
   dir2
   dir3

2. And you need to place a "watch" on these 3 directories for a specific file name pattern. Example: If a new file is added with name: watchme_9192.log, then you java logic should kick in and process that file.

So, based on that assumption, you can try: jnotify

JNotify is a java library that allow java application to listen to file system events, such as:
File created
File modified
File renamed
File deleted

Also, related: best practice for directory polling

来源：https://stackoverflow.com/questions/12870606/directory-scanner-in-java