How to deal with a very large text file?

Question

I\'m currently writing something that needs to handle very large text files (a few GiB at least). What\'s needed here (and this is fixed) is:

• CSV-based, following

Answer 1

• Finding the start of line:

Stick with UTF-8 and \n denoting the end of the line should not be a problem. Alternatively you can allow UTF-16, and recognize the data: it has to be quoted (for instance), has N commans (semicolons) and another end of line. Can read the header to know how many columns the structure.

• Inserting into the middle of the file

can be achieved by reserving some space at the end/beginning of each line.

• appending lines at the end

That's trivial as long as the file is locked (as any other modifications)