data-compression

I want to compress big text files with python (I am talking about >20Gb files). I am not any how an expert so I tried to gather the info I found and the following seems to work : import bz2 with open('bigInputfile.txt', 'rb') as input: with bz2.BZ2File('bigInputfile.txt.bz2', 'wb', compresslevel = 9) as output: while True: block = input.read(900000) if not block: break output.write(block) input.close() output.close() I am wondering if this syntax is correct and if there is a way to optimize it ? I have an impression that I am missing something here. Many thanks. Your script seems correct, but

I haven't seen any documentation on the topic, but that doesn't mean it doesn't exist. zaph Yes, compress the data with zlib. @Brad Larson posted on this: see here and added the code as well. There is a CocoaPod which uses Objective-Zip by flyingdolphinstudio. Following @Zaph & @Brad Larson's posts, below are the 2 methods gzipInflate and gzipDeflate that work just fine to compress/decompress NSData . (code reformatted from cocoadev.com/wiki/NSDataCategory #import "zlib.h" // don't forget to add libz.1.2.x.dylib into your project - (NSData *)gzipInflate:(NSData*)data { if ([data length] == 0)

Is there a standard way to do this? Googling -- "approximate entropy" bits -- uncovers multiple academic papers but I'd like to just find a chunk of pseudocode defining the approximate entropy for a given bit string of arbitrary length. (In case this is easier said than done and it depends on the application, my application involves 16,320 bits of encrypted data (cyphertext). But encrypted as a puzzle and not meant to be impossible to crack. I thought I'd first check the entropy but couldn't easily find a good definition of such. So it seemed like a question that ought to be on StackOverflow!

Is there anyway in .Net (C#) to extract data from a zip file without decompressing the complete file? Simply I possibly want to extract data (file) from the start of a zip file, obviously this depends if the compression algorithm compress the file in a deterministic order. DotNetZip is your friend here. As easy as: using (ZipFile zip = ZipFile.Read(ExistingZipFile)) { ZipEntry e = zip["MyReport.doc"]; e.Extract(OutputStream); } (you can also extract to a file or other destinations). Reading the zip file's table of contents is as easy as: using (ZipFile zip = ZipFile.Read(ExistingZipFile)) {