使用Boost计算ECMA-128 64位CRC

Question

With the following program I'm trying to compute a 64bit CRC as per the ECMA-128 standard .

The test data is "123456789" and I'm trying to match the same data provided here which suggests that the result for CRC-64/ECMA-182 should be 62ec59e3f1a4f00a. Unfortunately I get 9d13a61c0e5b0ff5 which is the CRC-64/WE result.

I started with the sample code provided here . I created the 64 bit hashes with the normal polynomial representation of 0x42F0E1EBA9EA3693 for the ECMA-128 64bit crc.

---

I get the following VS warning: C4293: '<<': shift count negative or too big, undefined behavior. It is for this macro:

BOOST_STATIC_CONSTANT( least, sig_bits = (~( ~(least( 0u )) << Bits )) );

As far as I can tell the 0 is being bit shifted by the full range of 64 bits which is undefined behavior. I'm surprised I don't see this warning for the 32bit crc.

---

How can this program be corrected to calculate an ECMA-128 64bit crc correctly without undefined behavior?

// from https://www.boost.org/doc/libs/1_67_0/libs/crc/crc.html#usage
#include <boost/crc.hpp>      // for boost::crc_basic, boost::crc_optimal
#include <boost/cstdint.hpp>  // for boost::uint16_t
#include <algorithm>  // for std::for_each
#include <cassert>    // for assert
#include <cstddef>    // for std::size_t
#include <iostream>   // for std::cout
#include <ostream>    // for std::endl

//#define SHOW_ERROR

#if defined( SHOW_ERROR )
#define CRC ecma_crc // expected
#else 
#define CRC other_crc // actually received
#endif

int main()
{
    // This is "123456789" in ASCII
    unsigned char const  data[] = {0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39};
    std::size_t const    data_len = sizeof(data) / sizeof(data[0]);

    // The expected CRC for the given data
    boost::uint16_t const  expected = 0x29B1;
    // Expected CRCs for "123456789" as per https://www.nitrxgen.net/hashgen/
    long long const  other_crc = 0x9D13A61C0E5B0FF5; // Wolfgang Ehrhardt http://www.wolfgang-ehrhardt.de/crchash_en.html
    long long const  ecma_crc = 0x62EC59E3F1A4F00A; // CRC-64-ECMA-128 https://en.wikipedia.org/wiki/Cyclic_redundancy_check

    // Simulate CRC-CCITT
    boost::crc_basic<16>  crc_ccitt1(0x1021, 0xFFFF, 0, false, false);
    crc_ccitt1.process_bytes(data, data_len);
    assert(crc_ccitt1.checksum() == expected);

    // Repeat with the optimal version (assuming a 16-bit type exists)
    boost::crc_optimal<16, 0x1021, 0xFFFF, 0, false, false>  crc_ccitt2;
    crc_ccitt2 = std::for_each(data, data + data_len, crc_ccitt2);
    assert(crc_ccitt2() == expected);

    // Attempt 64 bit CRC
    boost::crc_basic<64>  crc_64_ecma1(0x42F0E1EBA9EA3693, 0xFFFFFFFFFFFFFFFF, 0, false, false);
    crc_64_ecma1.process_bytes(data, data_len);
    assert(crc_64_ecma1.checksum() == CRC);

    boost::crc_optimal<64, 0x42F0E1EBA9EA3693, 0xFFFFFFFFFFFFFFFF, 0, false, false>  crc_64_ecma2;
    crc_64_ecma2 = std::for_each(data, data + data_len, crc_64_ecma2);
    assert(crc_64_ecma2() == CRC);

    std::cout << "All tests passed." << std::endl;
    return 0;
}

Answer 1

Look carefully at 62ec59e3f1a4f00a and 9d13a61c0e5b0ff5, digit by digit. Do you see a relation between those two?

Update:

Good! You saw it. (I hate to deprive others the joy of discovery.) As to why, you can look to Greg Cook's CRC catalog for the definition of the ECMA-182 CRC, referred to as simply CRC-64 there:

width=64 poly=0x42f0e1eba9ea3693 init=0x0000000000000000 refin=false refout=false xorout=0x0000000000000000 check=0x6c40df5f0b497347 residue=0x0000000000000000 name="CRC-64"

This defines the CRC with an initial value of zero, and a final exclusive-or of zero, ie no change. This is not common for CRCs, since it results in a string of zeros of any length having the same CRC, equal to zero.

CRC-64/WE is also defined there, which gives:

width=64 poly=0x42f0e1eba9ea3693 init=0xffffffffffffffff refin=false refout=false xorout=0xffffffffffffffff check=0x62ec59e3f1a4f00a residue=0xfcacbebd5931a992 name="CRC-64/WE"

This is more common, with an initialization and exclusive-or both with all ones, since the CRC of a string of zeros will depend on the length of the string, and the CRC of a zero length message is zero.

However that disagrees with the calculation at the link you provided. The link you provided calculated it with an initial value of zero, whereas the definition in Greg's catalog initializes the CRC with all ones. Only one of them is correct.

Update 2:

Greg's catalog is correct. The linked on-line hash calculator is wrong. I verified the CRC definition by looking at Wolfgang Ehrhardt's (the "WE" in CRC-64/WE) source code (a zip file) . The linked calculator is doubly wrong, in that it gives the CRC-64/WE check value 62ec... as the CRC-64/ECMA-182 result, and it gives a result for CRC-64/WE that matches neither CRC-64. Beware of what you find on the interwebs.

Answer 2

Correct parameters for CRC

Boost has the following typedef for CRC-32/B/BZIP2/AAL5/DECT-B:

typedef crc_optimal<32, 0x04C11DB7, 0xFFFFFFFF, 0xFFFFFFFF, true, true>
  crc_32_type;

Similarly the ECMA-128 64bit CRC requires a final xor value (3rd parameter):

boost::crc_basic<64>  crc_64_ecma1(0x42F0E1EBA9EA3693, 0xFFFFFFFFFFFFFFFF, 0xFFFFFFFFFFFFFFFF, false, false);
                                                                           ^^^^^^^^^^^^^^^^^^

Reason for warning message

The following template

template < std::size_t Bits >
class crc_basic

uses

template < std::size_t Bits >
struct mask_uint_t

and this mask_uint_t has been specialized for various numbers of bits:

• unsigned char -> BOOST_STATIC_CONSTANT( least, sig_bits = (~( least(0u) )) );
• unsigned short -> BOOST_STATIC_CONSTANT( least, sig_bits = (~( least(0u) )) );
• unsigned int -> BOOST_STATIC_CONSTANT( least, sig_bits = (~( least(0u) )) );

But the unspecialized version is different and triggers the undefined behavior warning:

• BOOST_STATIC_CONSTANT( least, sig_bits = (~( ~(least( 0u )) << Bits )) );

On MSVC it still yields the correct value, but it should probably not be relied upon.