Writing to JSON file, then reading this same file and getting “JSONDecodeError: Extra data”

Question

I have a very large json file (9GB). I\'m reading in one object from it at a time, and then deleting key-value pairs in this object when the key is not in the list fields<

Answer 1

You are basically dumping new json objects into a file every time you are calling json.dump(profile, f). But that does not generate valid JSON, since it does not emped the objects correctly. E.g. {}{} instead of {{},{}}

As for a solution - the size of your JSON makes reading / writing while holding everything in memory a bad solution. I would probably try the library https://pypi.org/project/jsonstreams/ or something like this.