How can I tail a zipped file without reading its entire contents?

Question

I want to emulate the functionality of gzcat | tail -n.

This would be helpful for times when there are huge files (of a few GB\'s or so). Can I tail the last few li

Answer 1

zindex creates and queries an index on a compressed, line-based text file in a time- and space-efficient way.

https://github.com/mattgodbolt/zindex

Answer 2

If you have control over what goes into the file in the first place, if it's anything like a ZIP file you could store chunks of predetermined size with filenames in increasing numerical order and then just decompress the last chunk/file.

Answer 3

Well, you can do that if you previously creates an index for each file ...

I've developed a command line tool which creates indexes for gzip files which allow for very quick random access inside them, and it does this interleaved with actions (extract, tail, continuous tail, etc): https://github.com/circulosmeos/gztool

But you can do a tail (-t), and the index will be automatically created: if you're gonna do the same in the future it'll be much quicker, and anyway the first time it will take the same time as a gunzip | tail:

$ gztool -t my_file.gz

Answer 4

BGZF is used to created index gzip compressed BAM files created by Samtools. These are randomly accessible.

http://samtools.sourceforge.net/

Answer 5

No, you can't. The zipping algorithm works on streams and adapts its internal codings to what the stream contains to achieve its high compression ratio.

Without knowing what the contents of the stream are before a certain point, it's impossible to know how to go about de-compressing from that point on.

Any algorithm which allows you to de-compress arbitrary parts of it will require multiple passes over the data to compress it.

Answer 6

If it's an option, then bzip2 might be a better compression algorithm to use for this purpose.

Bzip2 uses a block compression scheme. As such, if you take a chunk of the end of your file which you are sure is large enough to contain all of the last chunk, then you can recover it with bzip2recover.

The block size is selectable at the time the file is written. In fact that's what happens when you set -1 (or --fast) to -9 (or --best) as compression options, which correspond to block sizes of 100k to 900k. The default is 900k.

The bzip2 command line tools don't give you a nice friendly way to do this with a pipeline, but then given bzip2 is not stream oriented, perhaps that's not surprising.