File size limit for read()?

Question

I\'m running into a problem while trying to load large files using Python 3.5. Using read() with no arguments sometimes gave an OSError: Invalid argument<

Answer 1

Yes, you have bumped into a bug.

Good news is that someone else has also found it and already created an issue for it in the Python bug tracker, see: Issue24658 - open().write() fails on 2 GB+ data (OS X). This, seems, is platform depended (OS-X only) and is reproducible when using read and/or write. Apparently an issue exists with the way fread.c is implemented in the libc implementation for OS-X see here.

Bad News is that it is still open (and, currently, inactive) so, you'll have to wait until it is resolved. Either way, you can still take a look at the discussion there if you're interested for the specifics.

---

As a solution, I'm pretty sure you can side-step the issue until it is fixed by reading in chunks and chaining the chunks during processing. Do the same when writing. Unfortunate but, it might do the trick.