Independent

gaikwad.madhav@gmail.com

Applications and Real-Time base64 compression deflate encoding ba64 is a text encoding for binary data that is never larger than standard base64. An encoder races DEFLATE compression against plain base64 and emits whichever final text is shorter. Compressed output is marked by a leading "=" character, which is inside the base64 alphabet, so it survives every base64-safe channel, yet can never begin a valid base64 string, so the two forms are unambiguous. A CRC-32 over the decoded bytes guarantees that a ba64 decoder never silently returns wrong data. The plain form is byte-identical to base64, so ba64 decoders are a drop-in replacement wherever base64 is read today. An optional padding method decouples the emitted length from the compressibility of the input.

Introduction Base64 expands binary data by 33%. Applications that embed binary in JSON, cookies, logs, or size-limited message channels pay that tax even when the underlying bytes are highly compressible. Ad hoc solutions (gzip-then-base64) fork the wire format and lose base64 compatibility. ba64 addresses this with a single rule: emit compressed output only when it is strictly shorter than plain base64. The result is an encoding that is never larger than base64, is byte-identical to base64 in the common (incompressible) case, and is unambiguously self-describing. The emitted length depends on the compressibility of the input, which exposes the compression side channel of , summarized for TLS in , where an adversary controls part of an input carried alongside secret data. defines an OPTIONAL padding method that decouples the emitted length from compressibility, specifies the policy for selecting a pad, and states the limits of the mitigation. This document specifies version 1 of the ba64 format. Version 1 is intended to be frozen: evolution occurs only through new method identifiers assigned by a future revision.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. Throughout, "base64" means canonical base64 as defined in , Section 4: the alphabet A-Z a-z 0-9 + /, "=" padding REQUIRED, no whitespace, no other characters, and the unused trailing bits of the final symbol set to zero. There is no URL-safe variant of ba64 ().

Encoded Forms A ba64 text is an ASCII string in exactly one of two forms: Dispatch is by the first character: "=" selects the compressed form; anything else (including the empty string, which decodes to zero bytes) selects the plain form.

Frame Layout

decoded_len (unsigned LEB128) Little-endian base-128; each byte holds 7 value bits with the high bit as continuation. It MUST be minimally encoded (if longer than one byte, the final byte MUST NOT be 0x00), and MUST NOT exceed 9 bytes (so decoded_len < 2^63). Violations yield E_HEADER.

crc32 CRC-32/ISO-HDLC, the algorithm of gzip , zlib, and PNG: reflected polynomial 0xEDB88320, initial value 0xFFFFFFFF, final XOR 0xFFFFFFFF, reflected input and output. Stored little-endian, computed over the decoded (original) bytes, so it detects both channel corruption and decompressor divergence end-to-end. Anchors: CRC32("123456789") = 0xCBF43926, CRC32("") = 0.

Method Registry Method identifiers

ID	Meaning
0x00	Reserved, never valid
0x01	DEFLATE, raw ()
0x02	DEFLATE, raw, with padding ()
0x03 - 0xEF	Reserved for future revisions -> E_METHOD
0xF0 - 0xFF	Private use -> E_METHOD unless configured

Method 0x01 - Raw DEFLATE The payload is exactly one complete raw DEFLATE stream per , with no zlib or gzip wrapper. Normative outcomes:

• Inflater produces MORE than decoded_len bytes: abort at the cap, E_LENGTH_MISMATCH.
• Stream completes with FEWER than decoded_len bytes: E_LENGTH_MISMATCH.
• Payload exhausted before the final block completes, or malformed: E_PAYLOAD.
• Bytes remain after the final block: E_PAYLOAD.

Method 0x02 - Raw DEFLATE With Padding Method 0x02 is method 0x01 with one additional header field and a trailing pad. It allows an encoder to select the emitted text length independently of the compressibility of the input (, ).

• pad_len is an unsigned LEB128 under the same constraints as decoded_len: minimally encoded, at most 9 bytes; violations yield E_HEADER.
• pad is exactly pad_len bytes at the end of the frame. If fewer than pad_len bytes remain after pad_len itself, the result is E_TRUNCATED.
• payload is everything between pad_len and the pad: one complete raw DEFLATE stream, judged by the rules. The pad is removed BEFORE inflating and is never fed to the inflater.
• Encoders MUST write zero bytes for the pad. Decoders MUST NOT inspect the pad's content and MUST NOT reject a frame because of it. Pad content is therefore not integrity-protected: the CRC covers the decoded bytes only.
• pad_len = 0 is valid, and is grammatically distinct from a method-0x01 frame carrying the same payload.

A padded frame occupies the length its encoder selected. Padding therefore trades the compression saving for length uniformity.

Decoding Algorithm Input: string s, caller limit max_decoded_len (default 67,108,864). Checks MUST run in this order, so any invalid input yields one deterministic code: E_BASE64); return the bytes ("" -> empty bytes). 2. canonical-base64-decode s[1:] fail -> E_BASE64 3. read version (1 byte) missing -> E_TRUNCATED; !=0x01 -> E_VERSION 4. read method (1 byte) missing -> E_TRUNCATED; unknown -> E_METHOD 5. read decoded_len (LEB128) exhausted -> E_TRUNCATED; non-minimal or >9 bytes -> E_HEADER 6. decoded_len > max_decoded_len -> E_LIMIT_EXCEEDED (MUST precede any allocation proportional to decoded_len) 7. read crc32 (4 bytes) missing -> E_TRUNCATED 7b. method 0x02 only: read pad_len (LEB128) exhausted -> E_TRUNCATED; non-minimal or >9 bytes -> E_HEADER fewer than pad_len bytes remain -> E_TRUNCATED trim the trailing pad_len bytes; the payload is what is left 8. inflate payload with hard output cap = decoded_len (a cap of zero still caps) -> E_PAYLOAD / E_LENGTH_MISMATCH 9. CRC32(output) != crc32 -> E_CHECKSUM 10. return output ]]> E_TRUNCATED refers only to the fixed header fields (steps 3-7b); truncation inside the DEFLATE stream is E_PAYLOAD. A decoder MUST NOT reject a frame merely because a conforming encoder would have chosen the plain form: decoders validate the grammar, not encoder optimality.

Error Taxonomy Conformance means the exact code, not merely "an error." Error codes

Code	Fires when
E_BASE64	input or frame body is not canonical base64
E_TRUNCATED	frame ends inside the header, or is shorter than its pad
E_HEADER	LEB128 non-minimal or longer than 9 bytes
E_VERSION	version byte != 0x01
E_METHOD	method unknown, reserved, or unconfigured
E_LIMIT_EXCEEDED	claimed decoded_len exceeds the caller's limit
E_PAYLOAD	DEFLATE stream malformed, short, or has trailing bytes
E_LENGTH_MISMATCH	inflated size != decoded_len
E_CHECKSUM	CRC-32 of output != stored value

Implementation note: the boundary between E_PAYLOAD and E_LENGTH_MISMATCH for a truncated DEFLATE stream depends on the underlying inflate library (some treat premature end of input as end of stream). Both codes reject the frame and the CRC still prevents silently-wrong output; callers SHOULD treat the two as equivalent "corrupt frame" outcomes.

Encoding Requirements

1. Output MUST be one of the two forms, using canonical base64 throughout.
2. Floor rule: the compressed form MUST be emitted only if its final text length is strictly less than the plain form's; ties take plain. Consequently len(ba64(x)) <= len(base64(x)) for every input.
3. decoded_len MUST equal the true input length, minimally encoded; crc32 MUST be over the input bytes.
4. An encoder MAY skip compression entirely. An encoder that always emits plain base64 is degenerate but fully conforming.
5. DEFLATE level is the encoder's choice (SHOULD default to 6). Encoded output is therefore NOT canonical: the same input may yield different valid texts. Any equality, deduplication, cache-keying, or MAC comparison MUST operate on the decoded bytes, never on ba64 text.
6. Padding is OPTIONAL: an encoder MAY emit method 0x02 (). The floor rule still binds. A decoder MUST accept method 0x02 whether or not the encoder ever produces it.

The Padding Policy Let n be the input length and P = 4*ceil(n/3) the plain-form length. A compressed text is 1 + 4k characters and is therefore congruent to 1 modulo 4, while P is congruent to 0 modulo 4. A compressed text can never equal the plain-form length, and the greatest length the floor rule permits is P - 3. The RECOMMENDED policy takes one parameter, a quantum Q, a positive multiple of 4 in characters:

1. Build the method-0x02 frame with pad_len = 0; call its text length L0.
2. Target T = Q*ceil((L0 - 1)/Q) + 1, clamped down to P - 3.
3. If T < L0 the compressed form cannot fit under the floor: emit the plain form.
4. Otherwise pad the frame to F = 3*(T - 1)/4 bytes. Frame lengths F, F - 1 and F - 2 all encode to T characters, so an encoder that cannot hit F exactly (the pad_len varint changes size as the pad grows) uses whichever of the three it can reach.

Q = 4 pads by at most three bytes. A quantum greater than or equal to P yields P - 3 characters for every input that fits under the floor rule, so the emitted length is then a function of n alone. One bit remains observable at every quantum: whether the input compressed enough to fit under the floor rule. An encoder that must not leak that bit uses the always-plain mode, whose length is a function of n alone.

Limits and Resource Safety Decoders MUST enforce max_decoded_len before any allocation proportional to the claimed size (step 6) and MUST cap inflation at decoded_len (step 8). A cap of zero MUST still cap: some inflate APIs treat an output limit of 0 as unlimited, so a frame claiming decoded_len = 0 whose payload hides megabytes MUST fail after producing at most one byte. A 100-byte frame claiming 8 GiB costs O(header) work to reject.

Security Considerations Integrity is not authenticity: CRC-32 detects accidental corruption, not tampering. Where malice is in scope, verify a MAC or signature over the decoded bytes at the application layer. Compression side channel (CRIME/BREACH class, , ): where the emitted length is observable to an adversary who controls part of an input carried alongside secret data, compression leaks the secret through the length. Implementations MUST NOT ba64-compress attacker-influenced data concatenated with secret data in such settings; the always-plain encoder mode applies there, and its length, 4*ceil(n/3), is a function of the input length alone. Where compression cannot be disabled, the padding of and quantizes the emitted length. At quantum Q, a one-byte change in compressibility is unobservable unless it crosses a bucket boundary; at Q >= P all inputs of a given length that fit under the floor rule emit P - 3 characters. Padding increases the number of observations an attack requires. It does not remove the channel: bucket boundaries are still crossed under repeated queries, and the bit identified in still leaks. Padding is therefore a secondary control and not a substitute for separating attacker-influenced data from secret data. The pad is not covered by the CRC-32, which is computed over the decoded bytes. An attacker able to rewrite the text can rewrite the pad undetected, as with any other part of an unauthenticated message. The decoded output is unaffected. Decompression bombs are handled structurally by the limit and the hard inflate cap. Decoder error messages MUST NOT echo payload or decoded content by default: inputs may be secrets and errors end up in logs; the code and an offset suffice. Canonical base64 plus minimal LEB128 plus the ordered checks of ensure every ba64 text has exactly one interpretation (specific bytes or one specific error). There is no decode-side malleability.

Interoperability and Migration Compatibility is one-directional: a ba64 decoder reads every legacy base64 string unchanged (the plain form is base64); a plain base64 decoder cannot read the compressed form. Therefore deploy ba64 decoders everywhere first (drop-in, no behavior change), enable compressing encoders per channel only once every reader decodes ba64, and note that reverting encoders to the always-plain mode restores universal readability instantly. All ba64 characters are safe verbatim in JSON strings and cookie values. In URL components, "+", "/", and "=" MUST be percent-encoded. Transport artifacts (MIME line-wrapping, trailing newlines) MUST be stripped by the caller before decoding.

Non-Goals No URL-safe or alternative alphabets, no streaming, no compression negotiation, no dictionary registry, no public methods beyond the two defined here. No version 2 is planned; the version byte is an escape hatch, not a roadmap.

IANA Considerations This document has no IANA actions. The method registry in this document is maintained by revisions of this document, not by IANA.

Normative References Informative References ekoparty Security Conference Google

Golden Examples Every conforming decoder MUST reproduce these exactly. "" (empty plain form) "SGVsbG8sIHdvcmxkIQ==" -> "Hello, world!" (13 bytes) "=AQECDg4XTQECAP3/SGk=" -> "Hi" (stored block) "=AQEAAAAAAAEAAP//" -> "" (decoded_len 0) "=AQICDg4XTQABAgD9/0hp" -> "Hi" (0x02, pad_len 0) "=AQICDg4XTQMBAgD9/0hpAAAA" -> "Hi" (0x02, pad_len 3) "=AQICDg4XTQQBAgD9/0hp/wDerQ==" -> "Hi" (0x02, nonzero pad) ]]> Error-direction anchors: E_TRUNCATED "==" -> E_BASE64 "SGVsbG8" -> E_BASE64 (padding) "QR==" -> E_BASE64 (bits) "SGVs bG8=" -> E_BASE64 (space) version=2 -> E_VERSION varint 80 00 -> E_HEADER claimed 1TB -> E_LIMIT_EXCEEDED pad_len past end of frame -> E_TRUNCATED "=AQEAAAAAAAEBAP7/WA==" -> E_LENGTH_MISMATCH (zero-cap trap) ]]>

Conformance Vectors The normative conformance corpus is a set of machine-readable JSON vector files (decode passthrough, valid frames of both methods, error cases with required codes, encoder invariants, resource-exhaustion frames, and a large differential set). They are the executable form of this specification: an implementation conforms if and only if it reproduces every decode vector's bytes exactly and raises every error vector's exact code. Six independent implementations (Python, TypeScript, Go, Rust, Java, C#) are maintained against this corpus and agree pairwise over the full differential set. The corpus is published alongside this document.

Acknowledgments The format's testing discipline follows the SQLite and zlib approach: many more lines of tests and vectors than of codec, continuous fuzzing (), and every historical finding frozen into the corpus.