The NTCF Network and Telemetry Compression Format

Introduction

Problem Security and network telemetry is high in volume, highly repetitive, and almost always interrogated along a small number of dimensions: source and destination IP address, autonomous system number (ASN), country, port, event type, and time. Operators retain large archives and incur two costs: storage of the data at rest, and the time spent decompressing whole archives to answer a single question during an incident. General-purpose compressors (gzip, zstd, lz4, xz) reduce the storage cost but produce an opaque blob: answering "which records involved 203.0.113.5?" requires full decompression, and the format offers no analytics. General-purpose columnar analytics formats such as Apache Parquet make data queryable but are not specialised for telemetry semantics (for example, IP, ASN, and CIDR types and IP-range pruning) nor for crash-safe streaming append from an edge sensor.

Goals NTCF aims to be, simultaneously:

Compact -- competitive with or better than the best general-purpose compressors on representative telemetry, by encoding meaning rather than bytes.
Searchable in place -- equality search and a useful subset of analytical queries answerable without decompressing the whole file, using embedded zone maps, Bloom filters, and optional inverted indexes.
Streaming- and crash-safe -- appendable from a long-running sensor such that a process crash leaves a readable file containing all committed records.
Self-describing and versioned -- a file carries its own schema and a format version that gates compatibility.

Non-Goals (This Version) Cryptographic authentication and encryption of files, distributed query, joins across files, and a network storage service are out of scope for format version 1. See .

Conventions and Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. Data type conventions used throughout this document:

All multi-octet integers are little-endian unless stated otherwise.
u8, u16, u32, and u64 denote unsigned integers of that width in bits.
uvarint denotes an unsigned LEB128 variable-length integer: seven bits of payload per octet, with the most significant bit of each octet set on every octet except the last.
varint denotes a signed integer encoded as ZigZag(value) followed by uvarint, where ZigZag maps a two's-complement 64-bit value v to (v left-shifted by 1) XOR (v arithmetic-right-shifted by 63).
The notation ceil(x) denotes the least integer not less than x.

Terminology:

Column: a named, typed sequence of one value per row.
Column chunk: the encoded, optionally compressed, self-validating serialization of one column within one segment.
Segment: a row group; a contiguous run of rows whose columns are stored as adjacent column chunks. A segment is located by absolute offset from the footer; it is not independently framed (see ).
Footer: the trailing metadata block: schema descriptor, segment and column directory, zone-map statistics, and file-level totals.
Zone map: the per-column, per-segment minimum and maximum of present values, used to prune segments.
Checkpoint footer: a footer written mid-stream by an appending writer to bound data loss on crash (see ).

Design Overview NTCF compresses in two cooperating layers. The semantic layer operates on typed columns and removes structural redundancy that a byte compressor cannot perceive: near-monotonic timestamps (delta-of-delta), low-cardinality enumerations (dictionary), repeated values (run-length), small-range integers (frame-of-reference bit packing), and general integers (variable-length plus ZigZag). An encoder SHOULD trial candidate encodings per column chunk and keep the smallest, and MUST always include a baseline (Plain or Raw) so the chosen encoding is never larger than the baseline. The entropy layer is a conventional byte compressor -- zstd, lz4, or none -- applied per column chunk to the semantically encoded octets. The two layers are complementary: the semantic layer maps a repeated IP column to small dictionary ordinals; the entropy layer removes the residual byte-level redundancy. A file is a fixed header, a sequence of segments (each an opaque concatenation of column chunks and optional index blobs), and a footer. The footer is read first for fast open and for predicate pruning. Crash recovery relies on checkpoint footers and a backward scan (), and NOT on per-segment framing.

Segments and Column Chunks

Segments A segment is the concatenation of one column chunk per schema column, in schema column order, with each indexed column's chunk OPTIONALLY followed by its index blob (). A segment has no magic and no self-contained header; its extent and the location of every chunk and index within it are given by the footer's segment directory (). All chunks in a segment encode the same number of rows.

Column Chunk A column chunk is self-validating: it carries its own checksum and the lengths needed to bound decompression. To decode a chunk a reader MUST: (1) verify checksum over stored; (2) enforce the decompression limits of against uncompressedLen and the ratio uncompressedLen divided by storedLen; (3) entropy-decode stored to exactly uncompressedLen octets; (4) semantic-decode rows values; and (5) if a presence bitmap is present, apply it to mark null rows.

Presence Bitmap (Nullability) When a column contains null (absent) values, the chunk stores a presence bitmap of ceil(rows/8) octets. Bit i (least significant bit first within each octet, that is, octet i divided by 8, bit i modulo 8) is set when row i is present (non-null). The encoded value stream contains one value per row; the value at a null row is a placeholder (zero for integers, empty for byte values) and MUST be ignored by readers when the presence bit is clear. Storing a placeholder per null row, rather than only present values, is a deliberate simplification of format version 1; placeholders compress well under run-length and dictionary encodings. A future version MAY define a present-values-only encoding.

Semantic Encodings Columns are mapped to one of two physical domains. The integer domain carries values as u64; the byte domain carries variable-length octet strings. The mapping from logical type to domain is given in . All integer encodings are exact over the full u64 range because their arithmetic is performed modulo 2^64 identically on encode and decode. encodingID values are stable and assigned as follows: A reader MUST reject a chunk whose encodingID is unknown for its kind. The number of values produced MUST equal rows.

Bit Packing Bit packing serialises a sequence of unsigned integers using exactly width bits each, least significant bit first, with no per-value octet alignment. width is in the range 0 to 64. A width of 0 encodes a sequence of zeros and occupies no octets. The total size is ceil((count times width) divided by 8) octets. The Bitpack frame-of-reference encoding subtracts a per-chunk minimum before packing; the dictionary encodings pack ordinal indices at width equal to the number of bits needed to represent dictLen minus 1.

Dictionary Encoding A dictionary chunk has the following layout:

dictLen (uvarint): the number of distinct values.
The value table, in ascending sorted order. For DictInt: the first value as uvarint, then each subsequent value as the uvarint non-negative gap from its predecessor. For DictBytes: per entry, a uvarint length followed by that many octets.
width (u8): bits per ordinal, equal to the number of bits needed to represent dictLen minus 1.
The per-row ordinals, bit-packed at width bits ().

Each ordinal MUST be strictly less than dictLen.

Entropy Compression The entropy layer is applied to the semantically encoded octets of a chunk. A writer SHOULD select none when entropy compression would not reduce size. compressionID values: For compressionID 2, selector 0 means the remaining octets are the uncompressed semantic octets (used when the data is incompressible); selector 1 means the remaining octets form an LZ4 block (not an LZ4 frame) that decompresses to exactly uncompressedLen octets. For all codecs, a reader MUST verify that decompression yields exactly uncompressedLen octets and MUST treat any deviation as corruption. The zstd frame format used by compressionID 1 is specified in ; the LZ4 block format used by compressionID 2 is described in .

Indexes For each column marked indexed, a writer MAY emit an index blob immediately after that column's chunk within the segment; its location is recorded in the footer (indexOffset, indexLength). An indexLength of 0 means no index.

Bloom Filter The bit count m equals wordCount times 64. Bit b is located in word (b divided by 64) at bit position (b modulo 64). A value's membership uses double hashing of its XXH64 digest h (integers are hashed as their 8-octet little-endian form): with h1 equal to h and h2 equal to (h right-shifted by 33) bitwise-OR (h left-shifted by 31), and with h2 replaced by the constant 0x9E3779B97F4A7C15 if it would otherwise be 0, probe i for 0 le i less than k addresses bit (h1 + i times h2) modulo m. A writer SHOULD size the filter to the column's distinct cardinality at a target false-positive rate (the reference uses one percent). A clear probe is definitive non-membership; a fully set result is probabilistic.

Inverted Index Each entry is a key followed by a posting list. The key is a uvarint for integer keys, or a uvarint length plus that many octets for byte keys. The posting list is a uvarint bitmapLen followed by bitmapLen octets containing a Roaring Bitmap , per the Roaring Bitmap serialization specification, of the zero-based row positions within the segment that hold that key. Inverted indexes are OPTIONAL; when absent, equality is resolved by zone-map and Bloom pruning followed by a scan of the decoded column.

Footer

Schema Descriptor Each column descriptor is: nameLen (uvarint), name (octets), type (u8, see ), and flags (u8: bit0 = indexed, bit1 = nullable).

Footer Body Each segment directory entry is: offset (u64), length (u64), rows (uvarint), minTS (u64), maxTS (u64), colCount (uvarint, which MUST equal the schema column count), then one column directory entry per column:

Footer Trailer The footer body is immediately followed by footerLen (u32, the octet length of the footer body), crc32c (u32, CRC-32C over the footer body), and the trailer magic (bytes[4] equal to "NTCF"). To open a file, a reader reads the final 4 octets and verifies the trailer magic, reads footerLen and crc32c from the 8 octets preceding it, slices the footer body of footerLen octets immediately before, verifies the CRC, and parses the body. A reader MUST enforce that footerLen is at most 256 mebibytes () and MUST verify that the body lies wholly between the header and the trailer.

Durability and Crash Recovery A writer that appends records over time (streaming ingestion) SHOULD write a checkpoint footer at intervals. A checkpoint footer is a complete footer () written at the current end of file; it is appended, never overwritten. Because earlier footers are never modified, the most recently completed footer is always intact even if the process terminates while writing a subsequent segment. On read, if the trailing footer is missing or fails validation, a reader MAY recover by scanning backward from the end of file for an occurrence of the trailer magic and attempting to parse a footer ending there; the first (latest) candidate whose footerLen and CRC validate is the recovered footer. Records written after the last checkpoint but before termination are not recoverable; this is the correct durability boundary. Dead intermediate footers MAY be reclaimed by an out-of-band compaction step.

Reading Algorithm (Informative)

Validate the header ().
Locate and validate the footer (); on failure, optionally recover ().
Parse the schema and the segment and column directory.
For a predicate of the form "column OP value": normalise value into the column domain (); for each segment consult the column's zone map and, if value cannot lie in the relevant bound, skip the segment without reading its body; for equality on an indexed column consult the Bloom filter and skip on a clear result; if an inverted index is present take its posting list, otherwise decode the single column chunk and scan it.
Combine per-segment row sets across predicates (intersection for AND, union for OR), then aggregate or project.

A count of all rows with no predicate is answered from totalRows with no body read.

Resource Limits Because every length and offset in a file is attacker-controlled, a reader MUST gate every allocation derived from a file-supplied count by a finite ceiling before allocating, and MUST bound decompression. The reference implementation enforces, and this document RECOMMENDS, at least the following ceilings: A reader MUST reject any "count times width" computation that would overflow, and MUST reject any offset or length that falls outside the file.

Security Considerations NTCF files frequently originate from untrusted parties: partner feeds, tenant sensors, and attacker probes. Conforming readers MUST treat all input as hostile. No panics or unbounded work: for any input octets, a decoder MUST return a value or an error; it MUST NOT crash, allocate without bound, read outside the file, or loop indefinitely. The reference implementation enforces this property with fuzz testing across the header and footer parser, every encoding decoder, the entropy layer, the index parser, and the query parser. Decompression bombs: a reader MUST enforce both an absolute uncompressed ceiling and an expansion-ratio cap () before and during entropy decoding, and MUST verify that the decompressed length equals the declared length. Integrity, not authenticity: the CRC-32C checksums on the header and footer and the XXH64 checksums on chunks detect accidental corruption only; they are NOT message authentication codes. An adversary with write access can forge a valid-looking file. Format version 1 provides no confidentiality and no authenticity. Consumers requiring those properties MUST layer an authenticated or encrypted transport or storage mechanism beneath NTCF. An authenticated container is a candidate for a future version. Resource exhaustion: the limits of bound memory and CPU per file; operators ingesting many files SHOULD additionally bound concurrency.

IANA Considerations This document requests registration of the following media type, per , and a file extension.

Type name:: application
Subtype name:: vnd.ntcf
Required parameters:: none
Optional parameters:: none
Encoding considerations:: binary
Magic number(s):: the four octets 0x4E 0x54 0x43 0x46 ("NTCF") at offset 0, and the same four octets as the final four octets of a complete file
File extension(s):: .ntcf
Security considerations:: see of this document
Interoperability considerations:: the format is versioned ()
Published specification:: this document
Intended usage:: COMMON
Change controller:: The NTCF Authors

If a registry of NTCF encodingID, compressionID, or logical type values is desired, this document suggests an "NTCF Encodings" registry seeded with the assignments in Sections 7, 8, and 13, under a "Specification Required" policy.

Versioning and Interoperability The header version field gates on-disk compatibility. A reader MUST refuse a file whose version it does not implement. Within a supported version, a reader MUST reject unknown encodingID, compressionID, and logical type values rather than guess. Additive changes that do not alter the octet layout of existing structures (for example, a new encoding identifier) MAY be made within a version only if existing readers can still reject the new identifier safely; otherwise the version MUST be incremented.