How to deal with a very large text file?

Question

I\'m currently writing something that needs to handle very large text files (a few GiB at least). What\'s needed here (and this is fixed) is:

• CSV-based, following

Answer 1

Would it be possible to split the file in "subfiles" (of course you must not split it within one Utf-8 char)? Then you need some meta data for each of the subfiles (total number of chars, and total number of lines).

If you have this and the "subfiles" are relative small so that you can always load one compleatly then the handling becomes easy.

Even the editing becomes easy to, because you only need to update the "subfile" and its meta data.

If you would put it to the edge: then you could use a database and store one line per data base row. -- If this is a good idea strongly depends on your use case.