Skip to main content

Module mdb_shard

Module mdb_shard 

Source
Expand description

The XET mdb_shard binary reconstruction format — a faithful, minimal port of xet-core’s mdb_shard crate.

cas-serve advertises XET support, which means the reconstruction shard it returns from GetReconstructionInfo must be parseable by any stock xet-core client (its MDBShardFile reader / FileReconstructor). The bespoke JSON shard the original #390 commit produced is not the XET wire format. This module ports the real binary container — the .mdb file — so a shard built here is byte-compatible with upstream.

§Why a port, not a dependency

Upstream mdb_shard is edition = "2024" and pulls in xet_config / xet_runtime / utils (and, transitively, the libtorch-coupled tree). cas-serve deliberately stays out of that tree (no libtorch). The binary format itself is plain Write/Read + little-endian primitives, so it ports cleanly to edition 2021.

§Layout (ported from mdb_shard/src/shard_format.rs)

The on-disk shard is, in order:

  1. Header (MDBShardFileHeader, 48 B): 32-byte magic tag, version (u64 LE, =2), footer_size (u64 LE).
  2. File-Info section: one FileDataSequenceHeader + N FileDataSequenceEntry per file (in ascending file-hash order), then a single bookend header. Each entry is 48 B ([u64;4] hash + 4×u32).
  3. CAS-Info section: one CASChunkSequenceHeader + N CASChunkSequenceEntry per xorb (in ascending xorb-hash order), then a single bookend header.
  4. File lookup: (u64 truncated file hash, u32 index) per file.
  5. CAS lookup: (u64 truncated xorb hash, u32 index) per xorb.
  6. Chunk lookup: (u64 truncated chunk hash, u32 cas idx, u32 chunk offset) per chunk, sorted by key.
  7. Footer (MDBShardFileFooter, 200 B): section offsets/counts, the 32-byte chunk-hash HMAC key, timestamps, byte totals, and footer_offset.

truncate_hash(h) = h[0] (first u64 of the 4×u64 MerkleHash).

All multi-byte integers are little-endian, matching utils::serialization_utils.

Structs§

CasChunkSequenceEntry
CASChunkSequenceEntry (48 B): one chunk within a xorb.
CasChunkSequenceHeader
CASChunkSequenceHeader (48 B): opens a xorb’s chunk list.
FileDataSequenceEntry
FileDataSequenceEntry (48 B): one reconstruction segment — the xorb to pull chunks from, the inclusive-start / exclusive-end chunk range, and the total unpacked byte length of that range. Concatenating the chunk bytes for every segment, in segment order, reproduces the file.
FileDataSequenceHeader
FileDataSequenceHeader (48 B): the per-file record that opens its segment list. file_hash is the content address the shard is keyed by.
MdbCasInfo
One xorb’s content: its header + the ordered chunk entries it holds.
MdbFileInfo
One file’s reconstruction plan: its content-address and the ordered segments that rebuild it.
MdbShard
A parsed shard: header + footer, with helpers to seek into the byte buffer and reconstruct a file.
MdbShardBuilder
A builder that mirrors xet-core’s MDBInMemoryShard — accumulate file and xorb records, then serialize emits the .mdb byte stream in exactly the order MDBShardInfo::serialize_from does.
MdbShardFooter
200-byte shard footer carrying the section offsets, lookup counts, HMAC key, timestamps, and byte totals. Field order matches MDBShardFileFooter::serialize exactly.
MdbShardHeader
48-byte shard header: magic tag + version + footer size.

Constants§

MDB_ENTRY_SIZE
Every fixed-size File-Info / CAS-Info record is 48 bytes: a 32-byte hash ([u64;4]) plus four u32 slots. Asserted byte-for-byte against FileDataSequenceHeader, FileDataSequenceEntry, CASChunkSequenceHeader, and CASChunkSequenceEntry below.
MDB_FOOTER_SIZE
Size of the footer, in bytes: 21 u64 fields + the 32-byte HMAC key = 200. Verified against size_of::<MDBShardFileFooter>() upstream.
MDB_SHARD_HEADER_TAG
32-byte magic tag at the head of every shard file (shard_format.rs::MDB_SHARD_HEADER_TAG).

Functions§

build_shard
Build an MdbShardBuilder (plus, per xorb, its hash and concatenated raw bytes) from a chunk list, packing chunks into xorbs that respect the XET MAX_XORB_BYTES (64 MiB) and MAX_XORB_CHUNKS (8192) limits.
build_shard_with_cap
Like build_shard, but with a custom per-xorb byte cap (tests use a tiny cap to force multi-xorb aggregation without allocating 64 MiB).