Is it possible to efficiently insert or remove bytes from the middle of a large file, and if so how? Or am I stuck rewriting the entire file after the point
The most efficient way would be to seek to the position where you want to insert the element, read everything up to the end, insert the new element and copy back the rest.
The problem's not a language one, but actually how data is stored in media, where everything's just a long sequence of bits. You can imagine it as a single strip of paper with the data written out in pen. If you want to insert something, you'll have to push back everything that comes afterwards. Of course, if you've got a lots of empty space between blocks of data, you can insert your stuff in there (which is the idea behind Sparse Files), but that's hardly space-efficient.