如何将二进制数据编码为任意文本表示？

Question

I need a pair of functions for encoding binary data as any arbitrary text representation, and decoding it back

Say we have an ArrayBuffer of any size:

const buffer = new ArrayBuffer(1000)

Then we define a hexadecimal "lingo", and use it for encoding and decoding hex strings:

const lingo = "0123456789abcdef"

const text = encode(buffer, lingo)
const data = decode(text, lingo)

My goal is to define my own base48 "lingo", which omits vowels to avoid naughty words:

const lingo = "256789bcdfghjklmnpqrstvwxyzBCDFGHJKLMNPQRSTVWXYZ"

const text = encode(buffer, lingo)
const data = decode(text, lingo)

How can we approach creating the algorithms for efficiently transforming data between arbitrary representations? Even though this strikes me as something quite fundamental, I'm having a hard time finding resources to help me with this task

Bonus points if you can think of any plausible naughty words without any vowels, I even took out the numbers that look like vowels!

I'm working in javascript, but I'd also like to understand the principals in general. Thanks!

Answer 1

The challenge with streaming a series of bytes/digits and converting to another base is finding the most efficient ratio of source bytes/digits to target bytes/digits.

To determine the best ratio, the algorithm below contains a function dubbed mostEfficientChunk() which takes as parameters the source number base, the target number base, and the maximum source chunk size. This function then walks the source chunk sizes from 1 to the maximum chunk size, and determines the minimum number of bytes/digits required by the target number base. Eg, a source of Unit8Array whereby 1 byte ranges from 0 - 255 requires 3 bytes if converting to base 10. In this example, the efficiency then is measured at 1/3 or 33.33%. Then a source of 2 bytes is examined, which has a range of 0 - 65535 requiring 5 bytes of base 10, for an efficiency of 2/5 or 40%. So a source chunk size of 2 bytes when converting from base 256 to base 10 is more efficient than a chunk size of 1 byte. And so on, until the best ratio is found that is less than or equal to the maximum source chunk size.

The code below dumps the evaluation of mostEfficientChunk() to make the determination of the best chunk size readily apparent.

Then, once the chunk size is set, the source data is fed to 'code()' which queues up the source, and then if sufficient data exists to form a chunk, the function converts the chunk to the target base. Note that code() can be called continuously if the source is streaming. When the stream is finished, flush() must be called which appends digits that represent 0 until a chunk size is met, and then produces the final target chunk. Note that this last chunk is padded, so one will have to track the length of the original source to trim the decoding appropriately.

There are some comments and test cases in the code to help in understanding how the Encoder class operates.

 class EncodeStream { constructor( fromBase, toBase, encode = 'encode', maxChunkSize = 32 ) { console.assert( typeof fromBase === 'string' || typeof fromBase === 'number' ); console.assert( typeof toBase === 'string' || typeof toBase === 'number' ); console.assert( encode === 'encode' || encode === 'decode' ); this.encode = encode; if ( typeof fromBase === 'string' ) { this.fromBase = fromBase.length; this.fromBaseDigits = fromBase; } else { this.fromBase = fromBase |0; this.fromBaseDigits = null; } console.assert( 2 <= this.fromBase && this.fromBase <= 2**32 ); if ( typeof toBase === 'string' ) { this.toBase = toBase.length; this.toBaseDigits = toBase; } else { this.toBase = toBase |0; this.toBaseDigits = null; } console.assert( 2 <= this.toBase && this.toBase <= 2**32 ); if ( encode === 'encode' ) { this.chunking = this.mostEfficientChunk( this.fromBase, this.toBase, maxChunkSize ); } else { let temp = this.mostEfficientChunk( this.toBase, this.fromBase, maxChunkSize ); this.chunking = { bestSrcChunk: temp.bestTgtChunk, bestTgtChunk: temp.bestSrcChunk }; } console.log( `Best Source Chunk Size: ${this.chunking.bestSrcChunk}, Best Target Chunk Size: ${this.chunking.bestTgtChunk}` ); this.streamQueue = []; } code( stream ) { console.assert( typeof stream === 'string' || Array.isArray( stream ) ); if ( this.fromBaseDigits ) { this.streamQueue.push(...stream.split( '' ).map( digit => this.fromBaseDigits.indexOf( digit ) ) ); } else { this.streamQueue.push(...stream ); } let result = []; while ( this.chunking.bestSrcChunk <= this.streamQueue.length ) { // Convert the source chunk to a BigInt value. let chunk = this.streamQueue.splice( 0, this.chunking.bestSrcChunk ); let chunkValue = 0n; for ( let i = 0; i < chunk.length; i++ ) { chunkValue = chunkValue * BigInt( this.fromBase ) + BigInt( chunk[ i ] ); } // And now convert the BigInt value to a target chunk. let temp = new Array( this.chunking.bestTgtChunk - 1 ); for ( let i = 0; i < this.chunking.bestTgtChunk; i++ ) { temp[ this.chunking.bestTgtChunk - 1 - i ] = chunkValue % BigInt( this.toBase ); chunkValue = chunkValue / BigInt( this.toBase ); } result.push(...temp ); } // Finally, if the target base is represented by a string of digits, then map // the resulting array to the target digits. if ( this.toBaseDigits ) { result = result.map( digit => this.toBaseDigits[ digit ] ).join( '' ); } return result; } flush() { // Simply add zero digits to the stream until we have a complete chunk. if ( 0 < this.streamQueue.length ) { while ( this.streamQueue.length < this.chunking.bestSrcChunk ) { if ( this.fromBaseDigits ) { this.streamQueue.push( this.fromBaseDigits[ 0 ] ); } else { this.streamQueue.push( 0 ); } } } return this.code( this.fromBaseDigits? '': [] ); } mostEfficientChunk( sourceBase, targetBase, maxChunkSize ) { console.assert( 2 <= sourceBase && sourceBase <= 2 ** 32 ); console.assert( 2 <= targetBase && targetBase <= 2 ** 32 ); console.assert( 1 <= maxChunkSize && maxChunkSize <= 64 ); // Since BigInt does not have a LOG function, let's just brute force // determine the maximum number of target digits per chunk size of // source digits... let sBase = BigInt( sourceBase ); let tBase = BigInt( targetBase ); let mSize = BigInt( maxChunkSize ); let efficiency = 0; let result = { bestSrcChunk: 0, bestTgtChunk: 0 }; for ( let chunkSize = 1n; chunkSize <= mSize; chunkSize++ ) { let maxSrcValue = sBase ** chunkSize - 1n; let maxSrcBits = maxSrcValue.toString( 2 ).length; let d = 0n; let msv = maxSrcValue; while ( 0n < msv ) { msv = msv / tBase; d++; } if ( this.encode === 'encode' ) { console.log( `Source Chunk Size: ${chunkSize}, Max Source Value: ${maxSrcValue}\nTarget Chunk Size: ${d}, Max Target Value: ${tBase**d-1n}, Efficiency: ${Number( chunkSize * 10000n / d ) / 100}%` ); } if ( efficiency < Number( chunkSize ) / Number( d ) ) { efficiency = Number( chunkSize ) / Number( d ); result.bestSrcChunk = Number( chunkSize ); result.bestTgtChunk = Number( d ); } } return result; } } let source, toBase, encoder, encoderResult, decoder, decoderResult; source = [255,254,253,252,251]; toBase = '0123456789'; console.log( '\n\n' ); console.log( [ 'Encoding', source.join(','), `to base '${toBase}'` ] ); encoder = new EncodeStream( 256, toBase, 'encode', 2 ); encoderResult = ''; encoderResult += encoder.code( source ); encoderResult += encoder.flush(); console.log( `Encoded result: '${encoderResult}'` ); console.log( [ 'Decoding...' ] ); decoder = new EncodeStream( toBase, 256, 'decode', 2 ); decoderResult = ''; decoderResult += decoder.code( encoderResult ); decoderResult += decoder.flush(); console.log( `Decoded result: '${decoderResult}'` ); console.log( '\n\n' ); console.log( [ 'Encoding', source.join(','), `to base '${toBase}'` ] ); encoder = new EncodeStream( 256, toBase, 'encode', 16 ); encoderResult = ''; encoderResult += encoder.code( source ); encoderResult += encoder.flush(); console.log( `Encoded result: '${encoderResult}'` ); console.log( [ 'Decoding...' ] ); decoder = new EncodeStream( toBase, 256, 'decode', 16 ); decoderResult = ''; decoderResult += decoder.code( encoderResult ); decoderResult += decoder.flush(); console.log( `Decoded result: '${decoderResult}'` ); source = [255,254,253,252,251,250,249,248,247]; toBase = '256789bcdfghjklmnpqrstvwxyzBCDFGHJKLMNPQRSTVWXYZ'; console.log( '\n\n' ); console.log( [ 'Encoding', source.join(','), `to base '${toBase}'` ] ); encoder = new EncodeStream( 256, toBase, 'encode', 16 ); encoderResult = ''; encoderResult += encoder.code( source ); encoderResult += encoder.flush(); console.log( `Encoded result: '${encoderResult}'` ); console.log( [ 'Decoding...' ] ); decoder = new EncodeStream( toBase, 256, 'decode', 16 ); decoderResult = ''; decoderResult += decoder.code( encoderResult ); decoderResult += decoder.flush(); console.log( `Decoded result: '${decoderResult}'` );

Note that it appears you will need to open up the browser debugger to see the full console log results.