GC overhead limit exceeded with Apache POI

Question

I have 13 .xlsx files with about 1000 rows in each of them. Now I want to merge it to one .xlsx file with one sheet. I\'m using code from here http://blog.sodhanalibrary.com

Answer 1

POI is notoriously memory-hungry, so running out of memory is not uncommon when handling large Excel-files.

When you are able to load all original files and only get trouble writing the merged file you could try using an SXSSFWorkbook instead of an XSSFWorkbook and do regular flushes after adding a certain amount of content (see poi-documentation of the org.apache.poi.xssf.streaming-package). This way you will not have to keep the whole generated file in memory but only small portions.

Answer 2

Try allocating more memory eg.

java -Xmx8192m

Also what you can try is to merge in one xlsx file at a time instead of loading them all at once.

You can also move this line into your for loop:

excellFile1.close();

So you close it right away.

Answer 3

If you can avoid using the convenient but memory hungry workbook APIs, work instead with the streaming logic of processing data row by row, which is much more memory efficient.

In particular, pay particular attention to the usage of the: XSSFReader.SheetIterator for looping over the sheets.

And finally take a good look at the usage of the API: XSSFSheetXMLHandler. For processing the rows withing a sheet.

See the code on this project: https://github.com/jeevatkm/excelReader/blob/master/src/main/java/com/myjeeva/poi/ExcelReader.java

You define how you want to process each row by creating your own: new SheetContentsHandler....

This is quite like SAX parsing, it will not take a bit at your ram.

 private void readSheet(StylesTable styles, ReadOnlySharedStringsTable sharedStringsTable,
      InputStream sheetInputStream) throws IOException, ParserConfigurationException, SAXException {

    SAXParserFactory saxFactory = SAXParserFactory.newInstance();
    XMLReader sheetParser = saxFactory.newSAXParser().getXMLReader();

    ContentHandler handler =
        new XSSFSheetXMLHandler(styles, sharedStringsTable, sheetContentsHandler, true);

    sheetParser.setContentHandler(handler);
    sheetParser.parse(new InputSource(sheetInputStream));
  }

Answer 4

This issue occurs due to the below reason

The java.lang.OutOfMemoryError: GC overhead limit exceeded error is the JVM’s way of signalling that your application spends too much time doing garbage collection with too little result. By default the JVM is configured to throw this error if it spends more than 98% of the total time doing GC and when after the GC only less than 2% of the heap is recovered.

if you just want to neglect this issue you can set the following vm options:

-XX:-UseGCOverheadLimit

Refer link on GC overhead for more information.

You can also use the below switches to assign more heap memory to your application. Run a pilot on your application for some time and identify how much memory would be better for your application

-Xms128m -Xmx512m(these switches sets the initial heap memory size to 128mb and Max memory to 512mb)