python eval vs ast.literal_eval vs JSON decode

Question

I am converting 2 MB of data as a string into a dict. The input is serialized in JSON.

Anyways I am currently using ast.literal_eval and I get the dictionary I want,

Answer 1

Not an answer exactly, but it should be noted that eval and literal_eval are not the same thing. The ast.literal_eval won't run arbitrary code.

That said, I agree with using JSON; I just wanted to point out that eval != literal_eval

Answer 2

Yes, there's definitely a reason: eval() is evil. Your code might read untrusted data one day, an this would allow an attacker to run arbitrary code on your machine.

You shouldn't use ast.literal_eval() to decode JSON either. It cannot decode every valid JSON string and is not meant to be used for this purpose. Simply use json.loads(), it's reasonably fast.

Answer 3

I don't really like this attitude on stackoverflow (and elsewhere) telling people without any context that what they are doing is insecure and they shouldn't do it. Maybe it's just a throwaway script to import some data, in that case why not choose the fastest or most convenient way?

In this case, however, json.loads is not only more secure, but also more than 4x faster (depending on your data).

In [1]: %timeit json.loads(data)
10000 loops, best of 3: 41.6 µs per loop

In [2]: %timeit eval(data)
10000 loops, best of 3: 194 µs per loop

In [3]: %timeit ast.literal_eval(data)
1000 loops, best of 3: 269 µs per loop

If you think about it makes sense json is a such more constrained language/format than python, so it must be faster to parse with an optimized parser.

Answer 4

No. Unless you hit one of two scenarios:

1. That's not JSON!

   Someone puts __import__('os').system('rm -rf /') in the file instead. You are boned.

2. It's JSON, but not the Python-like part!

   Someone puts true, false, null, or a Unicode escape somewhere in it. Happy birthday.