My app scans part of a file system, and my users reported it was very slow when they were scanning a network drive. Testing my code, I identified the bottleneck: the methods
I faced exactly the same problem
The solution for our case was quite simple: since our directory structure was following a standard (there where no directory which had the '.' character in it's name), I just followed the standard, and applied a very simple heuristic: "in our case, directories doesn't have the '.' character in it's name". This simple heuristic reduced drastically the number of times our application had to call the isDirectory() function of the java.io.File class.
Maybe this is your case. Maybe on your directory structure you could know if a File is a directory just by it's naming conventions.