zlib inflate error : Z_DATA_ERROR while the received packets is out-of-order or lost

问题

I have work this for weeks, very hope for your help!!! please forgive my poor english.

First, I think it's necessary to describe the Application Scenario:

what data I want to decompress?----the data is come from the network traffic of the internet. In these traffic, there are some data are compressed by gzip and store in the http or tcp packet, if the data size is huge and large than the maxlength of tcp payload, it will be sliced and transmiss. I can extract the compressed data from these packet, and group these data by the tcp stream. so I can assure that the data extracted from these packets of one specific tcp stream is belong to Same data source. so the data is consist of many compressed data chunk, the Application Scenario require that you need to decompress the data immediately once recieved one packet. For each tcp stream, we maintain a z_stream data structure.
When does the program report an error? ----All of the error is "Z_DATA_ERROR: invalid distance too far back". then I find when the recieved packet is out-of-order or some packet is lossed, the error will happen!

One simple case:

Compressed data is split into multiple data blocks and stored in network data packets(p1, p2, p3, p4, p5, p6, p7), and then transmiss in one specific tcp stream. For each tcp stream, we maintain a z_stream data structure. Obviously, p1 include the gzip header 0x1f 0x8b 0x08...), but due to the Uncertainty in network transmission, the packet recieved may be out-of-order or loss, for example: (p1,p2,p5,p6,p7,p3,p4),the first two packet can decompree normally, but when decompress p5, the error occur(Z_DATA_ERROR).

SO, I have these problem:

Due to the application scenario, I need to decompress the data once recieved one packet with gzip content-encoding. So I want to know if zlib supports such a function----directly decompress a compressed block without having to consider the packet arrival order?
I also test the influce of packet recieved order: If I sort the data in its original order and then decompress it sequentially, it will decompress normally.
Thirdly, Logically speaking, for the packeted recieved order (p1,p2,p5,p6,p7,p3,p4), when decompress these packet sequentially, p1,p2 will decompress successfully, p5,p6,p7 will decompress failed, the next packet recieved is p3, Logically speaking, it should be decompress successfully, but when I test this case, it failed, I don't understand this.
I also found a confusing problem, which does not happen often: if I sort the packet as (p1,p2,p3,p5,p4...), Logically speaking, when decompress p5, it should report an error, buf it decompress sucessfully, I don't understand this.

the following is source code:


/**
 * buf: the gzip compressed data that extract form tcp packet
 */
void dowithGzipDataByZlib(z_stream * p_zlib_strm, unsigned char * buf, int buflen)
{
    int zlib_status = Z_OK;
    int bytes_dc_now = 0;
    unsigned char pNowResBuff[4096];
    printf("-------\n");
    (*p_zlib_strm).avail_in = buflen;
    (*p_zlib_strm).next_in = buf;
    do {
        memset(pNowResBuff,0,4096);
        (*p_zlib_strm).avail_out = 4096;
        (*p_zlib_strm).next_out = pNowResBuff;
        zlib_status = inflate (p_zlib_strm, Z_NO_FLUSH);
        printf("inflate status:%d\n",zlib_status);
        if(Z_OK != zlib_status && Z_STREAM_END!=zlib_status){
            printf("(*p_zlib_strm).avail_in:%d\n",(*p_zlib_strm).avail_in);
            printf("err msg:%s\n",p_zlib_strm->msg);
            return ;
        }
        bytes_dc_now = 4096 - (*p_zlib_strm).avail_out;
        // printf("bytes_dc_no:")
    } while(0 == (*p_zlib_strm).avail_out) ; 

    printf("(*p_zlib_strm).avail_in:%d\n",(*p_zlib_strm).avail_in);
}

// under the dirpath, there are some compressed data extract from the packets of one specific tcp stream, and store them in "file_basename_%d" file. (%d is the recieve order num: 1,2,3,4...)
void read( char* dirpath, char* file_basename)
{
    char filelist[99][255];
    int file_count = listDir(dirpath, filelist, 99, 255);
    char filepath[255];

    z_stream zlib_strm = {0};
    zlib_strm.zalloc = Z_NULL;
    zlib_strm.zfree = Z_NULL;
    zlib_strm.opaque = Z_NULL;
    zlib_strm.next_in = Z_NULL;
    zlib_strm.avail_in = 0;
    inflateInit2 (& zlib_strm, 32 | MAX_WBITS);

    FILE* fp;
    char buf[2048];

    // sort_file_ind: the array store the origin order of the compressed data.
    int sort_file_ind[99] = {0,1,2,3,15,16,17,18,19,20,21,4,5,6,7,8,9,10,11,12,13,14};

    for(int i=1;i<=file_count-2;i++)
    {
        memset(filepath,0,sizeof(filepath));
        // snprintf(filepath,sizeof(filepath), "%s%s%d",dirpath,file_basename,sort_file_ind[i]);
        snprintf(filepath,sizeof(filepath), "%s%s%d",dirpath,file_basename,i);
        printf("%s\n",filepath);
        fp = fopen(filepath,"r");
        if(fp == NULL){

            return;
        }
        fseek(fp, 0, SEEK_END);
        int flen = ftell(fp);
        fseek(fp, 0, SEEK_SET);
        memset(buf,0,sizeof(buf));
        int dlen = fread(buf, 1, flen, fp);
        if(dlen != flen){
            fclose(fp);
            return;
        }
        printf("dlen:%d\n",dlen);
        dowithGzipDataByZlib(&zlib_strm,(unsigned char *)buf,dlen);
        fclose(fp);
    }
}


char * dir = "/data/GzipDC/softDC/DocumentAnalyze/testbyzs/data/119.40.37.65.42050/";
char * base_filename = "119.40.37.65.42050>180.76.22.49.80_1_";

int main()
{
    read(dir,base_filename);
    return 0;
}

I've asked around and tried many things for days, and I really need someone with knowledge on the subject to weigh in here. Thanks for your time!

来源：https://stackoverflow.com/questions/60860604/zlib-inflate-error-z-data-error-while-the-received-packets-is-out-of-order-or

标签

compression

gzip

zlib

inflate