“OutOfMemoryError: GC overhead limit exceeded”: parse large json file with java

Question

I try to parse large json file (more 600Mo) with Java. My json file look like that:

{
    \"0\" : {\"link_id\": \"2381317\", \"overview\": \"mjklmk         


        

Answer 1

You have two choices:

1. Give more memory to the Java program by specifying the -Xmx argument, e.g. -Xmx1g to give it 1 Gb of memory.
2. Use a "streaming" JSON parser. This will scale to infinitely large JSON files.

json-simple has a streaming API. See https://code.google.com/p/json-simple/wiki/DecodingExamples#Example_5_-_Stoppable_SAX-like_content_handler

There are other libraries with good streaming parser, e.g. Jackson.