Split large files using python

Question

I have some trouble trying to split large files (say, around 10GB). The basic idea is simply read the lines, and group every, say 40000 lines into one file. But there are tw

Answer 1

For a 10GB file, the second approach is clearly the way to go. Here is an outline of what you need to do:

1. Open the input file.
2. Open the first output file.
3. Read one line from the input file and write it to the output file.
4. Maintain a count of how many lines you've written to the current output file; as soon as it reaches 40000, close the output file, and open the next one.
5. Repeat steps 3-4 until you've reached the end of the input file.
6. Close both files.