Is it possible to do CRC-32 calculation in splits?

Question

I use this trivial function to calculate the CRC checksum of a given file:

long i, j = 0;
int k = 0;
uint crc = 0xFFFFFFFF;
FileInfo file_info = new FileInfo(fil         


        

Answer 1

This equation is key:

CRC(a XOR b) == CRC(a) XOR CRC(b)

Suppose you want to compute the CRC of the following message:

"Always desire to learn something useful."

There exist functions to compute the CRC as:

crc_join(crc_part1("Always desire to lea"),
         crc_part2("rn something useful."))

If crc_part1 and crc_part2 zeros pad (\0) their arguments as shown, crc_join is just XOR.

crc_part1 = crc("Always desire to lea\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0")
crc_part2 = crc("\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0rn something useful.")

The trailing zeros can be accounted for in crc_part1 with lookup tables. The leading zeros can be ignored in crc_part2.

References:

1. High-speed Parallel Architecture for Software-based CRC
   Youngju. Do, Sung-Rok. Yoon, Taekyu. Kim, Kwang Eui. Pyun and Sin-Chong. Park
2. https://en.wikipedia.org/wiki/Cyclic_redundancy_check

Answer 2

I know this is an old question but this is what I use for "chunking" CRC calculations in case this helps anyone:

public static class Crc32Checksum
{
    private static readonly uint[] Table = GenerateTable();

    /// <summary>
    /// Calculates a CRC32 value for the data given.
    /// </summary>
    /// <param name="data">Data contents</param>
    /// <param name="offset">Byte offset to start reading</param>
    /// <param name="count">Number of bytes to read</param>
    /// <returns>The computed CRC32 value.</returns>
    public static int Calculate(byte[] data, int offset, int count) 
        => Calculate(0, data, offset, count);

    /// <summary>
    /// Calculates a new CRC32 value given additional data for the current CRC value.
    /// </summary>
    /// <param name="currentCrc">The current CRC value to start with</param>
    /// <param name="data">Additional data contents</param>
    /// <param name="offset">Byte offset to start reading</param>
    /// <param name="count">Number of bytes to read</param>
    /// <returns>The computed CRC32 value.</returns>
    public static int Calculate(int currentCrc, byte[] data, int offset, int count)
    {
        unchecked
        {
            uint crc = ~(uint)currentCrc;

            for (int i = offset, end = offset + count; i < end; i++)
                crc = (crc >> 8) ^ Table[(crc ^ data[i]) & 0xFF];

            return (int)~crc;
        }
    }

    private static uint[] GenerateTable()
    {
        unchecked
        {
            var table = new uint[256];

            const uint poly = 0xEDB88320;

            for (uint i = 0; i < table.Length; i++)
            {
                uint crc = i;

                for (int j = 8; j > 0; j--)
                {
                    if ((crc & 1) == 1)
                        crc = (crc >> 1) ^ poly;
                    else
                        crc >>= 1;
                }

                table[i] = crc;
            }

            return table;
        }
    }
}

Answer 3

It is indeed possible to parallelize the CRC-32 computation, but the details are messy and I'd need to spend about a day to come up with the code.

Let's look at the basic CRC algorithm, where there is no negation and no bit reversal.

For the string of bytes that you want to compute the CRC of, let's call it the message. The basic idea is that you treat the message as a polynomial in GF(2), and you compute its remainder modulo the CRC polynomial.

The basic CRC algorithm is additive/linear. If you have two messages a and b of the same length, then CRC(a XOR b) = CRC(a) XOR CRC(b).

Furthermore, if you pad the message on the right side with n zeros, the new CRC will be the old CRC times x^n mod the CRC polynomial.

 

With all that said, the only way to solve your problem is to really understand the mathematics behind the CRC algorithm and write your own custom code. Here's a long but very complete explanation of CRCs: http://www.ross.net/crc/download/crc_v3.txt

Answer 4

Yes. See crc32_combine_() in zlib for an example.