Iteratively parse JSON file

Question

I have 1000+ JSON files that look like

{
    \"name\": \"Some name\",
    \"part_num\": \"123456\",

    \"other_config\":         


        

Answer 1

This reminded me of XML back in the day, when we had three styles of parsers: DOM-based, event based (e.g., SAX), and lesser known pull-based (e.g., StAX). The latter two were more efficient, because they didn't require loading the entire file into memory when you only need bits of it.

Fortunately, similar things exist for JSON. For Python, take a look at yajl-py, which is a Python wrapper for the C-library Yajl.

It takes a bit more code to parse via an event-based parser, but it can be much more efficient.