Remove beginning of file without rewriting the whole file

Question

I have an embedded Linux system, that stores data in a very large file, appending new data to the end. As the file size grows near filling available storage space, I need to

Answer 1

You can achieve the goal with Linux kernel v3.15 above for ext4/xfs file system.

int ret = fallocate(fd, FALLOC_FL_COLLAPSE_RANGE, 0, 4096);

See here Truncating the first 100MB of a file in linux

Answer 2

What about setting up a separate process that renames the output file when it reaches a predefined size (for instance by adding the linux time at the end of the file name).

This would allow you to keep the old data and the main process will recreate the output file the next time it writes to it.

Another cron job may remove the old file every now and then.

Answer 3

The easiest solution for your old applications would be a FUSE filesystem which gives them access to the underlying file, but with the offset cyclically shifted. This would allow you to implement a ringbuffer at the physical level. The FUSE layer would be fairly trivial as it only needs to adjust all filepositions by a constant, modulo filesize.