问题
I have a Java application that parses pdf files in a directory and its subdirectories and creates a database using the information found in the files.
Everything was fine when I was using the program on around 900 files or so (which create a SQLite database with multiple tables, some of wihch contain 150k rows).
Now I'm trying to run my program on a larger set of data (around 2000 files) and at some point I get "OutOfMemoryError: Java Heap space". I changed the following line in my jdev.conf file:
AddVMOption -XX:MaxPermSize=256M
to 512M and I got the same error (though later, I think). I'm going to change it to something bigger again, but the thing is the computers this program will be used on are much older and thus don't have as much memory. Normally, the users are not going to add more than 30 files at a time, but I want to know at how many files I'm supposed to limit them to. Ideally, I'd like my program not to throw an error regardless of how many files are to be parsed.
At first, I thought it was my SQLite queries that were causing the error, but after reading up on Google, it's probably some recursive function. I isolated it (I think it's the correct one at least), to this function:
public static void visitAllDirsAndFiles(File dir) {
if(dir.isDirectory())
{
String[] children = dir.list();
for (int i=0; i<children.length; i++)
{
visitAllDirsAndFiles(new File(dir, children[i]));
}
}
else
{
try
{
BowlingFilesReader.readFile(dir);
}
catch(Exception exc)
{
exc.printStackTrace();
System.out.println("Other Exception in file: " + dir);
}
}
}
I think the problem might be that it recursively calls this function for each subsequent directory, but I'm really not sure that could be the problem. What do you think? If it might be, how can I make it so I don't get this error again? If you think it is impossible that this section alone causes the problem, I'll try to find which other part of the program can cause it.
The only other thing I can see causing that is that I connect to the database before calling the above method and I disconnect after it returns. The reason for that is that if I connect and disconnect after each file, my programs takes a lot longer to parse the data, so I'd really like not to have to change that.
回答1:
If the origin of the problem was recursion, you would get an error related to stack instead of heap. Seems that you have some kind of memory leak in BowlingFilesReader...
回答2:
MaxPermSize will only change your permagen space. You are running out of Heap space. Increase your max heap size with -Xmx properties
回答3:
I suggest you try increasing the heap space with something like
-mx1000m
If you have a 64-bit JVM you can use up to about 80% of the total memory of the machine. If you have a 32-bit JVM you may be limited to around 1200 to 1400 MB depending on the OS.
回答4:
BowlingFilesReader.readFile(dir); is suspicious. How much is it loading into memory, and why? If it's loading all files in a rather large directory into memory, that's an issue.
You also might try
java -Xmx 1G or more, depending on your RAM situation.
You could always try using a stack instead of a recursive function.
S = []
while( !S.isEmpty() ){
S.pop()
//operate
S.push( all of the current item's children )
}
回答5:
I think you should download a copy of the Memory Analyzer Tool MAT. Once you have that take a heap dump, load it in MAT, run the Leak Suspect report and you should be able to find out rather quickly what your issue is.
回答6:
@Adam Smith to your question(s)
The same problem happened... I'm going to close my ResultSets,
PreparedStatements and Statements now, but can you explain
why I have to close them? Don't they get de-allocated when
the method returns (thus they're no longer in the scope of any methods)?
most Jave IDE has built-in JProfiler or available plugin, integrate your project, run with profiler and then you'll see all Objects which are presents in Runtime, nothing complicated
then you have to close:
File I/O example here , JDBC Introduction (example on page's bottom), and to check and avoids to opening lots of Connections (not only JDBC Conn), create one and reuse that, if everything done you can Close this Conn too, (Connection is hard and slower action on both sides, on PC and Server too), all Streamed Object must be closed in The finally Block, because always works
as I mentioned these Object never gone from JVM UsedMemory and majorities ... never are GC'ed (for more details search on this forum), GC never works immediatelly
Runtime runtime = Runtime.getRuntime();
long total = runtime.totalMemory();
long free = runtime.freeMemory();
long max = runtime.maxMemory();
long used = total - free;
System.out.println(Math.round(max / 1e6) + " MB available before Cycle");
System.out.println(Math.round(total / 1e6) + " MB allocated before Cycle");
System.out.println(Math.round(free / 1e6) + " MB free before Cycle");
System.out.println(Math.round(used / 1e6) + " MB used before Cycle");
//.... your code with
//.....
runtime = Runtime.getRuntime();
long total = runtime.totalMemory();
long free = runtime.freeMemory();
long max = runtime.maxMemory();
long used = total - free;
System.out.println(Math.round(max / 1e6) + " MB available past Cycle");
System.out.println(Math.round(total / 1e6) + " MB allocated past Cycle");
System.out.println(Math.round(free / 1e6) + " MB free past Cycle");
System.out.println(Math.round(used / 1e6) + " MB used past Cycle");
runtime = Runtime.getRuntime();
runtime.gc();
//dealyed with some Timer ...
long total = runtime.totalMemory();
long free = runtime.freeMemory();
long max = runtime.maxMemory();
long used = total - free;
System.out.println(Math.round(max / 1e6) + " MB available after GC");
System.out.println(Math.round(total / 1e6) + " MB allocated after GC");
System.out.println(Math.round(free / 1e6) + " MB free after GC");
System.out.println(Math.round(used / 1e6) + " MB used after GC");
more infos on this forum and :-) described in English language :-)
来源：https://stackoverflow.com/questions/6878720/outofmemoryerror-java-heap-space-how-can-i-fix-this-error-that-happens-in-a-re