Iteratively parse JSON file

Question

I have 1000+ JSON files that look like

{
    \"name\": \"Some name\",
    \"part_num\": \"123456\",

    \"other_config\":         


        

Answer 1

YAJL is an event-driven parser with Python bindings. This way you can stop parsing once you have retrieved the info you need.

Answer 2

This reminded me of XML back in the day, when we had three styles of parsers: DOM-based, event based (e.g., SAX), and lesser known pull-based (e.g., StAX). The latter two were more efficient, because they didn't require loading the entire file into memory when you only need bits of it.

Fortunately, similar things exist for JSON. For Python, take a look at yajl-py, which is a Python wrapper for the C-library Yajl.

It takes a bit more code to parse via an event-based parser, but it can be much more efficient.

Answer 3

There is ijson which performs performs iterative json parsing in a pythonc way.

I guess that solution to your problem would be:

import ijson

f = open('...')
objects = ijson.items(f, 'other_config.item')
for part in objects:
    do_something_with(part)

Dislaimer I did not use that library.