Is it possible to efficiently insert or remove bytes from the middle of a large file, and if so how? Or am I stuck rewriting the entire file after the point
There is no way to insert data or remove data in O(1) in C# nor C++ nor any language with standard APIs or class libraries.
The best you could do is have some kind of file format that you define yourself, it could support O(1) insertions and removal. But you'd have to probably deal with fragmentation.
You could perhaps also look at an SQL database like sqlite which would take care of the complexities for you.