XET: Content-Addressable Storage Protocol for Efficient Data Transfer

1.1. Use Cases

XET is particularly well-suited for scenarios involving:¶

• Machine Learning: Model checkpoints often share common layers and parameters across versions, enabling significant storage savings through deduplication.¶

• Dataset Management: Large datasets with incremental updates benefit from chunk-level deduplication, where only changed portions need to be transferred.¶

• Version Control: Similar to Git LFS but with content-aware chunking that enables sharing across different files, not just versions of the same file.¶

• Content Distribution: The reconstruction-based model enables efficient range queries and partial downloads of large files.¶

2.1. Notational Conventions

All multi-byte integers in binary formats (xorb headers, shard structures) use little-endian byte order unless otherwise specified.¶

Hash values are 32 bytes (256 bits). When serialized, they are stored as raw bytes. When displayed as strings, they use a specific byte-swapped hexadecimal format (see Section 6.5).¶

Range specifications use different conventions depending on context:¶

3.1. Upload Flow

The upload process transforms files into content-addressed storage:¶

1. Chunking: Split files into variable-sized chunks using content-defined chunking (see Section 5).¶

2. Deduplication: Query for existing chunks to avoid redundant uploads (see Section 10).¶

3. Xorb Formation: Group new chunks into xorbs, applying compression (see Section 7).¶

4. Xorb Upload: Upload serialized xorbs to the CAS server.¶

5. Shard Formation: Create shard metadata describing file reconstructions.¶

6. Shard Upload: Upload the shard to register files in the system.¶

3.2. Download Flow

The download process reconstructs files from stored chunks:¶

1. Reconstruction Query: Request reconstruction information for a file hash.¶

2. Term Processing: Parse the ordered list of terms describing the file.¶

3. Data Fetching: Download required xorb ranges using provided URLs.¶

4. Chunk Extraction: Deserialize and decompress chunks from xorb data.¶

5. File Assembly: Concatenate chunks in term order to reconstruct the file.¶

4.1. Suite Definition

An algorithm suite specifies:¶

1. Content-Defined Chunking Algorithm: The rolling hash function and boundary detection logic used to split files into chunks.¶

2. Cryptographic Hash Function: The hash algorithm used for all content addressing (chunk hashes, xorb hashes, file hashes, verification hashes).¶

3. Keying Material: Domain separation keys for the hash function.¶

4. Algorithm Parameters: Chunk size bounds, mask values, lookup tables, and other constants.¶

4.2. Suite Requirements

Any conforming algorithm suite MUST satisfy:¶

• Determinism: Identical inputs MUST produce identical outputs across all implementations.¶

• Collision Resistance: The hash function MUST provide at least 128 bits of collision resistance.¶

• Preimage Resistance: The hash function MUST provide at least 128 bits of preimage resistance.¶

• Keyed Mode: The hash function MUST support keyed operation for domain separation.¶

4.3. Suite Negotiation

The algorithm suite used by an XET deployment is determined out-of-band, typically by the CAS server configuration. All clients interacting with a given server MUST use the same suite. Binary formats (xorbs, shards) do not contain suite identifiers; the suite is determined implicitly by the deployment context.¶

4.4. Defined Suites

This specification defines one algorithm suite:¶

• XET-GEARHASH-BLAKE3: Uses Gearhash for content-defined chunking and BLAKE3 for all cryptographic hashing. This is the default and currently only defined suite.¶

Future specifications MAY define additional suites with different algorithms.¶

5.1. Gearhash Algorithm

The XET-GEARHASH-BLAKE3 suite uses a Gearhash-based rolling hash algorithm [GEARHASH]. Gearhash maintains a 64-bit state that is updated with each input byte using a lookup table, providing fast and deterministic boundary detection.¶

5.2. Algorithm Parameters

The following constants define the chunking behavior for the XET-GEARHASH-BLAKE3 suite:¶

TARGET_CHUNK_SIZE  = 65536      # 64 KiB (2^16 bytes)
MIN_CHUNK_SIZE     = 8192       # 8 KiB (TARGET / 8)
MAX_CHUNK_SIZE     = 131072     # 128 KiB (TARGET * 2)
MASK               = 0xFFFF000000000000  # 16 one-bits

The Gearhash algorithm uses a lookup table of 256 64-bit constants. Implementations of the XET-GEARHASH-BLAKE3 suite MUST use the table defined in [GEARHASH] (see Appendix A for the complete lookup table).¶

5.3. Algorithm Description

The algorithm maintains a 64-bit rolling hash value and processes input bytes sequentially:¶

function chunk_file(data):
    h = 0                    # 64-bit rolling hash
    start_offset = 0         # Start of current chunk
    chunks = []

    for i from 0 to length(data):
        b = data[i]
        h = ((h << 1) + TABLE[b]) & 0xFFFFFFFFFFFFFFFF  # 64-bit wrap

        chunk_size = i - start_offset + 1

        # Skip boundary checks until minimum size reached
        if chunk_size < MIN_CHUNK_SIZE:
            continue

        # Force boundary at maximum size
        if chunk_size >= MAX_CHUNK_SIZE:
            chunks.append(data[start_offset : i + 1])
            start_offset = i + 1
            h = 0
            continue

        # Check for natural boundary
        if (h & MASK) == 0:
            chunks.append(data[start_offset : i + 1])
            start_offset = i + 1
            h = 0

    # Emit final chunk if any data remains
    if start_offset < length(data):
        chunks.append(data[start_offset : length(data)])

    return chunks

5.4. Boundary Rules

The following rules govern chunk boundary placement:¶

1. Boundaries MUST NOT be placed before MIN_CHUNK_SIZE bytes have been processed in the current chunk.¶

2. Boundaries MUST be forced when MAX_CHUNK_SIZE bytes have been processed, regardless of hash value.¶

3. Between minimum and maximum sizes, boundaries are placed when (h & MASK) == 0.¶

4. The final chunk MAY be smaller than MIN_CHUNK_SIZE if it represents the end of the file.¶

5. Files smaller than MIN_CHUNK_SIZE produce a single chunk.¶

5.5. Determinism Requirements

Implementations MUST produce identical chunk boundaries for identical input data. For the XET-GEARHASH-BLAKE3 suite, this requires:¶

• Using the exact lookup table values from Appendix A¶

• Using 64-bit wrapping arithmetic for hash updates¶

• Processing bytes in sequential order¶

• Applying boundary rules consistently¶

Other algorithm suites MUST specify their own determinism requirements.¶

5.6. Performance Optimization

Implementations MAY skip hash computation for the first MIN_CHUNK_SIZE - 64 - 1 bytes of each chunk, as boundary tests are not performed in this region.¶

This optimization does not affect output correctness because the Gearhash window is 64 bytes, ensuring the hash state is fully populated by the time boundary tests begin.¶

6.1. Chunk Hashes

Chunk hashes uniquely identify individual chunks based on their content. The algorithm suite determines how chunk hashes are computed.¶

For the XET-GEARHASH-BLAKE3 suite, chunk hashes use BLAKE3 keyed hash with DATA_KEY as the key:¶

DATA_KEY = {
  0x66, 0x97, 0xf5, 0x77, 0x5b, 0x95, 0x50, 0xde,
  0x31, 0x35, 0xcb, 0xac, 0xa5, 0x97, 0x18, 0x1c,
  0x9d, 0xe4, 0x21, 0x10, 0x9b, 0xeb, 0x2b, 0x58,
  0xb4, 0xd0, 0xb0, 0x4b, 0x93, 0xad, 0xf2, 0x29
}

function compute_chunk_hash(chunk_data):
    return blake3_keyed_hash(DATA_KEY, chunk_data)

6.2. Xorb Hashes

Xorb hashes identify xorbs based on their constituent chunks. The hash is computed using a Merkle tree construction where leaf nodes are chunk hashes. The Merkle tree construction is defined separately from the hash function.¶

INTERNAL_NODE_KEY = {
  0x01, 0x7e, 0xc5, 0xc7, 0xa5, 0x47, 0x29, 0x96,
  0xfd, 0x94, 0x66, 0x66, 0xb4, 0x8a, 0x02, 0xe6,
  0x5d, 0xdd, 0x53, 0x6f, 0x37, 0xc7, 0x6d, 0xd2,
  0xf8, 0x63, 0x52, 0xe6, 0x4a, 0x53, 0x71, 0x3f
}

{hash_hex} : {size}\n

function compute_internal_hash(children):
    buffer = ""
    for (hash, size) in children:
        buffer += hash_to_string(hash) + " : " + str(size) + "\n"
    return blake3_keyed_hash(INTERNAL_NODE_KEY, buffer.encode("utf-8"))

MEAN_BRANCHING_FACTOR = 4
MIN_CHILDREN = 2
MAX_CHILDREN = 2 * MEAN_BRANCHING_FACTOR + 1  # 9

function next_merge_cut(hashes):
    # hashes is a list of (hash, size) pairs
    # Returns the number of entries to merge (cut point)
    if length(hashes) <= 2:
        return length(hashes)

    end = min(MAX_CHILDREN, length(hashes))

    # Check indices [2, end) using 0-based indexing
    # Minimum merge is 3 children when input has more than 2 hashes
    for i from 2 to end:
        h = hashes[i].hash
        # Interpret last 8 bytes of hash as little-endian 64-bit unsigned int
        hash_value = bytes_to_u64_le(h[24:32])
        if hash_value % MEAN_BRANCHING_FACTOR == 0:
            return i + 1  # Cut after element i (include i+1 elements)

    return end

function merged_hash_of_sequence(hash_pairs):
    # hash_pairs is a list of (hash, size) pairs
    buffer = ""
    total_size = 0

    for (h, s) in hash_pairs:
        buffer += hash_to_string(h) + " : " + str(s) + "\n"
        total_size += s

    new_hash = blake3_keyed_hash(INTERNAL_NODE_KEY, buffer.encode("utf-8"))
    return (new_hash, total_size)

cfc5d07f6f03c29bbf424132963fe08d19a37d5757aaf520bf08119f05cd56d6 : 100

function compute_merkle_root(entries):
    # entries is a list of (hash, size) pairs
    if length(entries) == 0:
        return zero_hash()  # 32 zero bytes

    hv = copy(entries)

    while length(hv) > 1:
        write_idx = 0
        read_idx = 0

        while read_idx < length(hv):
            # Find the next cut point
            next_cut = read_idx + next_merge_cut(hv[read_idx:])

            # Merge this slice into one parent node
            hv[write_idx] = merged_hash_of_sequence(hv[read_idx:next_cut])
            write_idx += 1

            read_idx = next_cut

        hv = hv[0:write_idx]

    return hv[0].hash

function compute_xorb_hash(chunk_hashes, chunk_sizes):
    # Build leaf entries
    entries = []
    for i from 0 to length(chunk_hashes):
        entries.append((chunk_hashes[i], chunk_sizes[i]))

    # Compute root using the aggregated hash tree algorithm
    return compute_merkle_root(entries)

6.3. File Hashes

File hashes identify files based on their complete chunk composition. The computation is similar to xorb hashes, but with an additional final keyed hash step for domain separation.¶

For the XET-GEARHASH-BLAKE3 suite, file hashes use an all-zero key (ZERO_KEY) for the final hash:¶

ZERO_KEY = {
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
}

function compute_file_hash(chunk_hashes, chunk_sizes):
    # Build (hash, size) pairs for Merkle tree
    entries = zip(chunk_hashes, chunk_sizes)
    merkle_root = compute_merkle_root(entries)
    return blake3_keyed_hash(ZERO_KEY, merkle_root)

For empty files (zero bytes), there are no chunks, so compute_merkle_root([]) returns 32 zero bytes. The file hash is therefore blake3_keyed_hash(ZERO_KEY, zero_hash()), where zero_hash() is 32 zero bytes.¶

6.4. Term Verification Hashes

Term verification hashes are used in shards to prove that the uploader possesses the actual file data, not just metadata. The hash function is determined by the algorithm suite.¶

For the XET-GEARHASH-BLAKE3 suite, verification hashes use BLAKE3 keyed hash with VERIFICATION_KEY as the key:¶

VERIFICATION_KEY = {
  0x7f, 0x18, 0x57, 0xd6, 0xce, 0x56, 0xed, 0x66,
  0x12, 0x7f, 0xf9, 0x13, 0xe7, 0xa5, 0xc3, 0xf3,
  0xa4, 0xcd, 0x26, 0xd5, 0xb5, 0xdb, 0x49, 0xe6,
  0x41, 0x24, 0x98, 0x7f, 0x28, 0xfb, 0x94, 0xc3
}

The input is the raw concatenation of chunk hashes (not hex-encoded) for the term’s chunk range:¶

function compute_verification_hash(chunk_hashes, start_index, end_index):
    buffer = bytes()
    for i from start_index to end_index:  # end_index is exclusive
        buffer += chunk_hashes[i]  # 32 bytes each
    return blake3_keyed_hash(VERIFICATION_KEY, buffer)

6.5. Hash String Representation

When representing hashes as strings (e.g., in API paths), a specific byte reordering is applied before hexadecimal encoding.¶

function hash_to_string(hash):
    out = ""
    for segment in 0..4:                       # segments 0,1,2,3
        offset = segment * 8
        value  = little_endian_to_u64(hash[offset : offset + 8])
        out   += format("{:016x}", value)      # always 16 hex digits
    return out

function string_to_hash(hex_string):
    hash = []
    for segment in 0..4:
        start = segment * 16
        value = parse_u64_from_hex(hex_string[start : start + 16])
        hash.extend(u64_to_little_endian_bytes(value))
    return hash

Original hash bytes (indices 0-31):
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]

Reordered bytes:
[7, 6, 5, 4, 3, 2, 1, 0, 15, 14, 13, 12, 11, 10, 9, 8,
 23, 22, 21, 20, 19, 18, 17, 16, 31, 30, 29, 28, 27, 26, 25, 24]

String representation:
07060504030201000f0e0d0c0b0a090817161514131211101f1e1d1c1b1a1918

7.1. Size Constraints

MAX_XORB_SIZE   = 67108864  # 64 MiB maximum serialized size
MAX_XORB_CHUNKS = 8192      # Maximum chunks per xorb

Implementations MUST NOT exceed either limit. When collecting chunks:¶

1. Stop if adding the next chunk would exceed MAX_XORB_SIZE¶

2. Stop if the chunk count would exceed MAX_XORB_CHUNKS¶

3. Target approximately 1,024 chunks per xorb for typical workloads¶

7.2. Binary Format

Serialized xorbs have a footer so readers can locate metadata by seeking from the end:¶

+-------------------------------------------------------------+
|                 Chunk Data Region (variable)                |
|   [chunk header + compressed bytes repeated per chunk]      |
+-------------------------------------------------------------+
|                 CasObjectInfo Footer (variable)             |
+-------------------------------------------------------------+
|     Info Length (32-bit unsigned LE, footer length only)    |
+-------------------------------------------------------------+

The final 4-byte little-endian integer stores the length of the CasObjectInfo block immediately preceding it (the length does not include the 4-byte length field itself).¶

The chunk data region consists of consecutive chunk entries, each containing an 8-byte header followed by the compressed chunk data.¶

7.3. Chunk Header Format

Each chunk header is 8 bytes with the following layout:¶

7.4. Compression Schemes

Original:  [A0 A1 A2 A3 | B0 B1 B2 B3 | C0 C1 C2 C3 | ...]
Grouped:   [A0 B0 C0 ... | A1 B1 C1 ... | A2 B2 C2 ... | A3 B3 C3 ...]

function byte_group_4(data):
    n = length(data)
    groups = [[], [], [], []]

    for i from 0 to n:
        groups[i % 4].append(data[i])

    return concatenate(groups[0], groups[1], groups[2], groups[3])

function byte_ungroup_4(grouped_data, original_length):
    n = original_length
    base_size = n / 4
    remainder = n % 4

    # Calculate group sizes
    sizes = [base_size + (1 if i < remainder else 0) for i in range(4)]

    # Extract groups
    groups = []
    offset = 0
    for size in sizes:
        groups.append(grouped_data[offset : offset + size])
        offset += size

    # Interleave back to original order
    data = []
    for i from 0 to n:
        group_idx = i % 4
        pos_in_group = i / 4
        data.append(groups[group_idx][pos_in_group])

    return data

7.5. CasObjectInfo Footer

The metadata footer sits immediately before the 4-byte length trailer. Implementations MUST serialize fields in this exact order and reject unknown idents or versions.¶

8.1. Term Structure

Each term specifies:¶

• Xorb Hash: Identifies the xorb containing the chunks¶

• Chunk Range: Start (inclusive) and end (exclusive) indices within the xorb¶

• Unpacked Length: Expected byte count after decompression (for validation)¶

8.2. Reconstruction Rules

1. Terms MUST be processed in order.¶

2. For each term, extract chunks at indices [start, end) from the specified xorb.¶

3. Decompress chunks according to their compression headers.¶

4. Concatenate decompressed chunk data in order.¶

5. For range queries, apply offset_into_first_range to skip initial bytes.¶

6. Validate that the total reconstructed size matches expectations.¶

8.3. Range Queries

When downloading a byte range rather than the complete file:¶

1. The reconstruction API returns only terms overlapping the requested range.¶

2. The offset_into_first_range field indicates bytes to skip in the first term.¶

3. The client MUST truncate output to match the requested range length.¶

9.1. Overall Structure

+--------------------------------------------------------+
|                    Header (48 bytes)                   |
+--------------------------------------------------------+
|                    File Info Section                   |
|              (variable, ends with bookend)             |
+--------------------------------------------------------+
|                    CAS Info Section                    |
|              (variable, ends with bookend)             |
+--------------------------------------------------------+
|                   Footer (200 bytes)                   |
|                (omitted for upload API)                |
+--------------------------------------------------------+

9.2. Data Types

All multi-byte integers are little-endian. Field sizes are stated explicitly (e.g., “8-bit unsigned”, “32-bit unsigned”, “64-bit unsigned”). Hash denotes a 32-byte (256-bit) value.¶

9.3. Header

The header is 48 bytes at offset 0:¶

Offset  Size  Field
------  ----  -----
0       32    Tag (magic identifier)
32      8     Version (64-bit unsigned, MUST be 2)
40      8     Footer Size (64-bit unsigned, 0 if footer omitted)

The header version (2) and footer version (1) are independent version numbers that may evolve separately.¶

Offset  Size  Field
------  ----  -----
0       14    Application Identifier (ASCII, null-padded)
14      1     Null byte (0x00)
15      17    Magic sequence (fixed)

SHARD_MAGIC_SEQUENCE = {
  0x55, 0x69, 0x67, 0x45, 0x6a, 0x7b, 0x81, 0x57,
  0x83, 0xa5, 0xbd, 0xd9, 0x5c, 0xcd, 0xd1, 0x4a, 0xa9
}

HF_APPLICATION_ID = {
  0x48, 0x46, 0x52, 0x65, 0x70, 0x6f, 0x4d, 0x65,
  0x74, 0x61, 0x44, 0x61, 0x74, 0x61
}

9.4. File Info Section

The file info section contains zero or more file blocks, each describing a file reconstruction. The section ends with a bookend entry.¶

Offset  Size  Field
------  ----  -----
0       32    File Hash
32      4     File Flags (32-bit unsigned)
36      4     Number of Entries (32-bit unsigned)
40      8     Reserved (zeros)

Offset  Size  Field
------  ----  -----
0       32    CAS Hash (xorb hash)
32      4     CAS Flags (32-bit unsigned, reserved, MUST be set to 0)
36      4     Unpacked Segment Bytes (32-bit unsigned)
40      4     Chunk Index Start (32-bit unsigned)
44      4     Chunk Index End (32-bit unsigned, exclusive)

Offset  Size  Field
------  ----  -----
0       32    Range Hash (verification hash)
32      16    Reserved (zeros)

Offset  Size  Field
------  ----  -----
0       32    SHA-256 Hash of file contents
32      16    Reserved (zeros)

9.5. CAS Info Section

The CAS info section contains zero or more CAS blocks, each describing a xorb and its chunks. The section ends with a bookend entry.¶

Offset  Size  Field
------  ----  -----
0       32    CAS Hash (xorb hash)
32      4     CAS Flags (32-bit unsigned, reserved, MUST be set to 0)
36      4     Number of Entries (32-bit unsigned)
40      4     Num Bytes in CAS (32-bit unsigned, total uncompressed)
44      4     Num Bytes on Disk (32-bit unsigned, serialized xorb size)

Offset  Size  Field
------  ----  -----
0       32    Chunk Hash
32      4     Chunk Byte Range Start (32-bit unsigned)
36      4     Unpacked Segment Bytes (32-bit unsigned)
40      4     Flags (32-bit unsigned)
44      4     Reserved (32-bit unsigned, zeros)

function calculate_byte_range_starts(chunks):
    position = 0
    for chunk in chunks:
        chunk.byte_range_start = position
        position += chunk.unpacked_segment_bytes

Chunk 0: unpacked_segment_bytes = 1000
         byte_range_start = 0

Chunk 1: unpacked_segment_bytes = 2000
         byte_range_start = 1000

Chunk 2: unpacked_segment_bytes = 500
         byte_range_start = 3000

9.6. Footer

The footer is 200 bytes at the end of the shard. It is REQUIRED for stored shards but MUST be omitted when uploading shards via the upload API.¶

Offset  Size  Field
------  ----  -----
0       8     Version (64-bit unsigned, MUST be 1)
8       8     File Info Offset (64-bit unsigned)
16      8     CAS Info Offset (64-bit unsigned)
24      8     File Lookup Offset (64-bit unsigned)
32      8     File Lookup Num Entries (64-bit unsigned)
40      8     CAS Lookup Offset (64-bit unsigned)
48      8     CAS Lookup Num Entries (64-bit unsigned)
56      8     Chunk Lookup Offset (64-bit unsigned)
64      8     Chunk Lookup Num Entries (64-bit unsigned)
72      32    Chunk Hash Key
104     8     Shard Creation Timestamp (64-bit unsigned, Unix epoch seconds)
112     8     Shard Key Expiry (64-bit unsigned, Unix epoch seconds)
120     48    Reserved (zeros)
168     8     Stored Bytes on Disk (64-bit unsigned)
176     8     Materialized Bytes (64-bit unsigned)
184     8     Stored Bytes (64-bit unsigned)
192     8     Footer Offset (64-bit unsigned)

Total size: 200 bytes¶

Offset  Size  Field
------  ----  -----
0       8     Truncated File Hash (64-bit unsigned, first 8 bytes of hash)
8       4     File Info Entry Index (32-bit unsigned)

Offset  Size  Field
------  ----  -----
0       8     Truncated CAS Hash (64-bit unsigned, first 8 bytes of hash)
8       4     CAS Info Entry Index (32-bit unsigned)

Offset  Size  Field
------  ----  -----
0       8     Truncated Chunk Hash (64-bit unsigned, first 8 bytes of hash)
8       4     CAS Entry Index (32-bit unsigned)
12      4     Chunk Index within CAS (32-bit unsigned)

10.1. Local Session Deduplication

Within a single upload session, implementations SHOULD track chunk hashes to avoid processing identical chunks multiple times.¶

10.2. Cached Metadata Deduplication

Implementations MAY cache shard metadata locally to enable deduplication against recently uploaded content without network queries.¶

10.3. Global Deduplication

The global deduplication API enables discovering existing chunks across the entire storage system.¶

10.4. Fragmentation Prevention

Aggressive deduplication can fragment files across many xorbs, harming read performance. Implementations SHOULD:¶

• Prefer longer contiguous chunk ranges over maximum deduplication¶

• Target minimum run lengths (e.g., 8 chunks or 1 MiB) before accepting deduplicated references¶

11.1. Authentication

All API requests require authentication via Bearer token in the Authorization header:¶

Authorization: Bearer <access_token>

Tokens have associated scopes:¶

• read: Required for reconstruction and global deduplication queries¶

• write: Required for xorb and shard uploads (includes read permissions)¶

Token acquisition is provider-specific and outside the scope of this specification.¶

11.2. Common Headers

Request headers:¶

• Authorization: Bearer token (required)¶

• Content-Type: application/octet-stream for binary uploads¶

• Range: Byte range for partial requests (optional)¶

Response headers:¶

• Content-Type: application/json or application/octet-stream¶

11.3. Get File Reconstruction

Retrieves reconstruction information for downloading a file.¶

GET /v1/reconstructions/{file_id}

Path Parameters:¶

• file_id: File hash as hex string (see Section 6.5)¶

Optional Headers:¶

• Range: bytes={start}-{end}: Request specific byte range (end inclusive)¶

Response (200 OK):¶

{
  "offset_into_first_range": 0,
  "terms": [
    {
      "hash": "<xorb_hash_hex>",
      "unpacked_length": 263873,
      "range": {
        "start": 0,
        "end": 4
      }
    }
  ],
  "fetch_info": {
    "<xorb_hash_hex>": [
      {
        "range": {
          "start": 0,
          "end": 4
        },
        "url": "https://...",
        "url_range": {
          "start": 0,
          "end": 131071
        }
      }
    ]
  }
}

Response Fields:¶

• offset_into_first_range: Bytes to skip in first term (for range queries)¶

• terms: Ordered list of reconstruction terms¶

• fetch_info: Map from xorb hash to fetch information¶

Fetch Info Fields:¶

• range: Chunk index range this entry covers¶

• url: Pre-signed URL for downloading xorb data¶

• url_range: Byte range within the xorb for HTTP Range header (end inclusive). The start offset is always aligned to a chunk header boundary, so clients can parse chunk headers sequentially from the start of the fetched data.¶

Error Responses:¶

• 400 Bad Request: Invalid file_id format¶

• 401 Unauthorized: Invalid or expired token¶

• 404 Not Found: File does not exist¶

• 416 Range Not Satisfiable: Invalid byte range¶

11.4. Query Chunk Deduplication

Checks if a chunk exists in the system for deduplication.¶

GET /v1/chunks/default-merkledb/{chunk_hash}

Path Parameters:¶

• chunk_hash: Chunk hash as hex string (see Section 6.5)¶

Response (200 OK): Shard format binary (see Section 9)¶

Response (404 Not Found): Chunk not tracked by global deduplication¶

11.5. Upload Xorb

Uploads a serialized xorb to storage.¶

POST /v1/xorbs/default/{xorb_hash}

Path Parameters:¶

• xorb_hash: Xorb hash as hex string (see Section 6.5)¶

Request Body: Serialized xorb (binary, see Section 7)¶

Response (200 OK):¶

{
  "was_inserted": true
}

The was_inserted field is false if the xorb already existed; this is not an error.¶

Error Responses:¶

• 400 Bad Request: Hash mismatch or invalid xorb format¶

• 401 Unauthorized: Invalid or expired token¶

• 403 Forbidden: Insufficient token scope¶

11.6. Upload Shard

Uploads a shard to register files in the system.¶

POST /v1/shards

Request Body: Serialized shard without footer (binary, see Section 9)¶

Response (200 OK):¶

{
  "result": 0
}

Result values:¶

• 0: Shard already exists¶

• 1: Shard was registered¶

Error Responses:¶

• 400 Bad Request: Invalid shard format or referenced xorb missing¶

• 401 Unauthorized: Invalid or expired token¶

• 403 Forbidden: Insufficient token scope¶

12.1. Step 1: Chunking

Split each file into chunks using the algorithm in Section 5.¶

For each chunk:¶

1. Compute the chunk hash (see Section 6.1)¶

2. Record the chunk data, hash, and size¶

12.2. Step 2: Deduplication

For each chunk, attempt deduplication in order:¶

1. Local Session: Check if chunk hash was seen earlier in this session¶

2. Cached Metadata: Check local shard cache for chunk hash¶

3. Global API: For eligible chunks, query the global deduplication API¶

Record deduplication results:¶

• New chunks: Will be included in xorbs¶

• Deduplicated chunks: Record existing xorb hash and chunk index¶

12.3. Step 3: Xorb Formation

Group new (non-deduplicated) chunks into xorbs:¶

1. Collect chunks maintaining their order within files¶

2. Form xorbs targeting ~64 MiB total size¶

3. Compute compression for each chunk¶

4. Compute xorb hash for each xorb (see Section 6.2)¶

12.4. Step 4: Xorb Serialization and Upload

For each new xorb:¶

1. Serialize using the format in Section 7¶

2. Upload via POST to /v1/xorbs/default/{xorb_hash}¶

3. Verify successful response¶

All xorbs MUST be uploaded before proceeding to shard upload.¶

12.5. Step 5: Shard Formation

Build the shard structure:¶

1. For each file, construct file reconstruction terms¶

2. Compute verification hashes for each term (see Section 6.4)¶

3. Compute file hash (see Section 6.3)¶

4. Compute SHA-256 of raw file contents¶

5. Build CAS info blocks for new xorbs¶

12.6. Step 6: Shard Upload

1. Serialize the shard without footer¶

2. Upload via POST to /v1/shards¶

3. Verify successful response¶

12.7. Ordering and Concurrency

The following ordering constraints apply:¶

• All xorbs referenced by a shard MUST be uploaded before the shard¶

• Chunk computation for a file must complete before xorb formation¶

• Xorb hash computation must complete before shard formation¶

Within these constraints, operations MAY be parallelized:¶

• Multiple files can be chunked concurrently¶

• Multiple xorbs can be uploaded concurrently¶

• Deduplication queries can run in parallel¶

13.1. Step 1: Query Reconstruction

Request reconstruction information:¶

GET /v1/reconstructions/{file_id}
Authorization: Bearer <token>

For range queries, include the Range header:¶

Range: bytes=0-1048575

13.2. Step 2: Parse Response

Extract from the response:¶

• offset_into_first_range: Bytes to skip in first term¶

• terms: Ordered list of terms to process¶

• fetch_info: URLs and ranges for downloading data¶

13.3. Step 3: Download Xorb Data

For each term:¶

1. Look up fetch_info by xorb hash¶

2. Find fetch_info entry covering the term’s chunk range¶

3. Make HTTP GET request to the URL with Range header¶

4. Download the xorb byte range¶

Multiple terms may share fetch_info entries; implementations SHOULD avoid redundant downloads.¶

13.4. Step 4: Extract Chunks

For each downloaded xorb range:¶

1. Parse chunk headers sequentially¶

2. Decompress chunk data according to compression type¶

3. Extract chunks for the term’s index range¶

13.5. Step 5: Assemble File

1. For the first term, skip offset_into_first_range bytes¶

2. Concatenate extracted chunks in term order¶

3. For range queries, truncate to requested length¶

4. Write to output file or buffer¶

13.6. Caching Recommendations

See Section 14 for comprehensive caching guidance. Key recommendations:¶

• Cache decompressed chunks by hash for reuse across files and sessions¶

• Avoid caching reconstruction API responses (pre-signed URLs expire quickly)¶

• Cache shard metadata for local deduplication during uploads¶

13.7. Error Handling

Implementations SHOULD implement:¶

• Retry logic with exponential backoff for transient failures¶

• Validation of decompressed chunk sizes against headers¶

• Hash verification of reconstructed files when possible¶

14.1. Content Immutability

Objects in XET are identified by cryptographic hashes of their content. This content-addressable design provides a fundamental property: content at a given hash never changes. A xorb with hash H will always contain the same bytes, and a chunk with hash C will always decompress to the same data.¶

This immutability enables aggressive caching:¶

• Cached xorb data never becomes stale¶

• Cached chunk data can be reused indefinitely¶

• Cache invalidation is never required for content objects¶

The only time-sensitive elements are authentication tokens and pre-signed URLs, which are discussed separately below.¶

14.2. Client-Side Chunk Caching

Implementations SHOULD cache decompressed chunk data to avoid redundant decompression and network requests. The chunk hash provides a natural cache key.¶

14.3. Xorb Data Caching

Raw xorb data (compressed chunks with headers) MAY be cached by clients or intermediaries.¶

14.4. Shard Metadata Caching

Shard metadata enables deduplication without network queries. Implementations SHOULD cache shards from recent uploads for local deduplication.¶

14.5. Pre-Signed URL Handling

The reconstruction API returns pre-signed URLs for downloading xorb data. These URLs have short expiration times (typically minutes to hours) and MUST NOT be cached beyond their validity period.¶

Implementations MUST:¶

• Use URLs promptly after receiving them¶

• Re-query the reconstruction API if URLs have expired¶

• Never persist URLs to disk for later sessions¶

Reconstruction responses SHOULD be treated as ephemeral and re-fetched when needed rather than cached.¶

14.6. HTTP Caching Headers

Cache-Control: public, immutable, max-age=<url_ttl_seconds>
ETag: "<xorb_hash>"

Cache-Control: private, no-store

Cache-Control: private, max-age=3600
Vary: Authorization

14.7. CDN Integration

XET deployments typically serve xorb data through CDNs. The content-addressable design is well-suited for CDN caching:¶

• Hash-based URLs enable cache key stability¶

• Immutable content eliminates cache invalidation complexity¶

• Range requests enable partial caching of large xorbs¶

14.8. Proxy and Intermediary Considerations

Corporate proxies and other intermediaries MAY cache XET traffic.¶

Pre-signed URLs include authentication in the URL itself, allowing unauthenticated intermediaries to cache responses. However, reconstruction API requests include Bearer tokens in the Authorization header and SHOULD NOT be cached by intermediaries (the private directive prevents this).¶

15.1. Content Integrity

XET provides content integrity through cryptographic hashing:¶

• Chunk hashes verify individual chunk integrity¶

• Xorb hashes verify complete xorb contents¶

• File hashes verify complete file reconstruction¶

Implementations SHOULD verify hashes when possible, particularly for downloaded content.¶

15.2. Authentication and Authorization

Token-based authentication controls access to storage operations. Implementations MUST:¶

• Transmit tokens only over TLS-protected connections¶

• Avoid logging tokens¶

• Implement token refresh before expiration¶

• Use minimum required scope (prefer read over write)¶

15.3. Global Deduplication Privacy

The keyed hash protection in global deduplication prevents enumeration attacks:¶

• Servers never reveal raw chunk hashes¶

• Clients can only match chunks they possess¶

• The chunk hash key rotates periodically, and shard expiry limits the reuse window¶

15.4. Access-Controlled Content

XET deployments may support access-controlled or “gated” content, where users must be authorized (e.g., by accepting terms of service or requesting access) before downloading certain files. This has several implications for XET implementations.¶

15.5. Denial of Service Considerations

Large file uploads could exhaust server resources. Servers SHOULD implement:¶

• Rate limiting on API endpoints¶

• Maximum shard size limits (64 MiB)¶

• Maximum xorb size limits (64 MiB)¶

Normative References

[BLAKE3]

Aumasson, J., Neves, S., O'Connor, J., and Z. Wilcox-O'Hearn, "BLAKE3: One function, fast everywhere", 9 January 2020, <https://github.com/BLAKE3-team/BLAKE3-specs/blob/master/blake3.pdf>.

[LZ4]

Collet, Y., "LZ4 Frame Format Description", 2015, <https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md>.

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/rfc/rfc2119>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

Informative References

[FASTCDC]

Feng, D., Hu, Y., Hua, Y., Jiang, H., Liu, Q., Xia, W., Zhang, Y., and Y. Zhou, "FastCDC: A Fast and Efficient Content-Defined Chunking Approach for Data Deduplication", USENIX ATC 2016 , 2016, <https://www.usenix.org/conference/atc16/technical-sessions/presentation/xia>.

[GEARHASH]

Rijsdijk, S., "rust-gearhash: Fast, SIMD-accelerated GEAR hashing", 2020, <https://github.com/srijs/rust-gearhash>.

[MERKLE]

Merkle, R. C., "A Digital Signature Based on a Conventional Encryption Function", CRYPTO 1987, LNCS 293, pp. 369-378 , 1987.

B.1. Chunk Hash Test Vector

Input (ASCII): Hello World!
Input (hex): 48656c6c6f20576f726c6421

Hash (raw hex, bytes 0-31):
  a29cfb08e608d4d8726dd8659a90b9134b3240d5d8e42d5fcb28e2a6e763a3e8

Hash (XET string representation):
  d8d408e608fb9ca213b9909a65d86d725f2de4d8d540324be8a363e7a6e228cb

B.2. Hash String Conversion Test Vector

The XET hash string format interprets the 32-byte hash as four little-endian 64-bit unsigned values and prints each as 16 hexadecimal digits.¶

Hash bytes [0..31]:
  00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
  10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f

Expected XET string:
  07060504030201000f0e0d0c0b0a090817161514131211101f1e1d1c1b1a1918

The conversion formula:¶

function hash_to_string(h):
    # h is a 32-byte array
    result = ""
    for i from 0 to 4:
        start = i * 8
        end = (i + 1) * 8
        u64_bytes = slice(h, start, end)
        u64_val = little_endian_to_u64(u64_bytes)
        result += format("{:016x}", u64_val)
    return result

B.3. Internal Node Hash Test Vector

Child 1:
  hash (XET string): c28f58387a60d4aa200c311cda7c7f77f686614864f5869eadebf765d0a14a69
  size: 100

Child 2:
  hash (XET string): 6e4e3263e073ce2c0e78cc770c361e2778db3b054b98ab65e277fc084fa70f22
  size: 200

Buffer being hashed (ASCII, with literal \n newlines):
  c28f58387a60d4aa200c311cda7c7f77f686614864f5869eadebf765d0a14a69 : 100\n
  6e4e3263e073ce2c0e78cc770c361e2778db3b054b98ab65e277fc084fa70f22 : 200\n

Result (XET string):
  be64c7003ccd3cf4357364750e04c9592b3c36705dee76a71590c011766b6c14

B.4. Verification Range Hash Test Vector

Input: Two chunk hashes from test vector B.3, concatenated as raw bytes (not XET string format).¶

Chunk hash 1 (raw hex):
  aad4607a38588fc2777f7cda1c310c209e86f564486186f6694aa1d065f7ebad

Chunk hash 2 (raw hex):
  2cce73e063324e6e271e360c77cc780e65ab984b053bdb78220fa74f08fc77e2

Concatenated input (64 bytes, raw hex):
  aad4607a38588fc2777f7cda1c310c209e86f564486186f6694aa1d065f7ebad
  2cce73e063324e6e271e360c77cc780e65ab984b053bdb78220fa74f08fc77e2

Verification hash (XET string):
  eb06a8ad81d588ac05d1d9a079232d9c1e7d0b07232fa58091caa7bf333a2768

B.5. Reference Files

Complete reference files including sample chunks, xorbs, and shards are available at: https://huggingface.co/datasets/xet-team/xet-spec-reference-files¶