Expand description
The XET mdb_shard binary reconstruction format — a faithful, minimal port
of xet-core’s mdb_shard crate.
cas-serve advertises XET support, which means the reconstruction shard it
returns from GetReconstructionInfo must be parseable by any stock xet-core
client (its MDBShardFile reader / FileReconstructor). The bespoke JSON
shard the original #390 commit produced is not the XET wire format. This
module ports the real binary container — the .mdb file — so a shard built
here is byte-compatible with upstream.
§Why a port, not a dependency
Upstream mdb_shard is edition = "2024" and pulls in xet_config /
xet_runtime / utils (and, transitively, the libtorch-coupled tree).
cas-serve deliberately stays out of that tree (no libtorch). The binary
format itself is plain Write/Read + little-endian primitives, so it
ports cleanly to edition 2021.
§Layout (ported from mdb_shard/src/shard_format.rs)
The on-disk shard is, in order:
- Header (
MDBShardFileHeader, 48 B): 32-byte magic tag,version(u64 LE, =2),footer_size(u64 LE). - File-Info section: one
FileDataSequenceHeader+ NFileDataSequenceEntryper file (in ascending file-hash order), then a single bookend header. Each entry is 48 B ([u64;4]hash + 4×u32). - CAS-Info section: one
CASChunkSequenceHeader+ NCASChunkSequenceEntryper xorb (in ascending xorb-hash order), then a single bookend header. - File lookup:
(u64 truncated file hash, u32 index)per file. - CAS lookup:
(u64 truncated xorb hash, u32 index)per xorb. - Chunk lookup:
(u64 truncated chunk hash, u32 cas idx, u32 chunk offset)per chunk, sorted by key. - Footer (
MDBShardFileFooter, 200 B): section offsets/counts, the 32-byte chunk-hash HMAC key, timestamps, byte totals, andfooter_offset.
truncate_hash(h) = h[0] (first u64 of the 4×u64 MerkleHash).
All multi-byte integers are little-endian, matching
utils::serialization_utils.
Structs§
- CasChunk
Sequence Entry CASChunkSequenceEntry(48 B): one chunk within a xorb.- CasChunk
Sequence Header CASChunkSequenceHeader(48 B): opens a xorb’s chunk list.- File
Data Sequence Entry FileDataSequenceEntry(48 B): one reconstruction segment — the xorb to pull chunks from, the inclusive-start / exclusive-end chunk range, and the total unpacked byte length of that range. Concatenating the chunk bytes for every segment, in segment order, reproduces the file.- File
Data Sequence Header FileDataSequenceHeader(48 B): the per-file record that opens its segment list.file_hashis the content address the shard is keyed by.- MdbCas
Info - One xorb’s content: its header + the ordered chunk entries it holds.
- MdbFile
Info - One file’s reconstruction plan: its content-address and the ordered segments that rebuild it.
- MdbShard
- A parsed shard: header + footer, with helpers to seek into the byte buffer and reconstruct a file.
- MdbShard
Builder - A builder that mirrors xet-core’s
MDBInMemoryShard— accumulate file and xorb records, thenserializeemits the.mdbbyte stream in exactly the orderMDBShardInfo::serialize_fromdoes. - MdbShard
Footer - 200-byte shard footer carrying the section offsets, lookup counts, HMAC
key, timestamps, and byte totals. Field order matches
MDBShardFileFooter::serializeexactly. - MdbShard
Header - 48-byte shard header: magic tag + version + footer size.
Constants§
- MDB_
ENTRY_ SIZE - Every fixed-size File-Info / CAS-Info record is 48 bytes: a 32-byte hash
(
[u64;4]) plus fouru32slots. Asserted byte-for-byte againstFileDataSequenceHeader,FileDataSequenceEntry,CASChunkSequenceHeader, andCASChunkSequenceEntrybelow. - MDB_
FOOTER_ SIZE - Size of the footer, in bytes: 21
u64fields + the 32-byte HMAC key = 200. Verified againstsize_of::<MDBShardFileFooter>()upstream. - MDB_
SHARD_ HEADER_ TAG - 32-byte magic tag at the head of every shard file
(
shard_format.rs::MDB_SHARD_HEADER_TAG).
Functions§
- build_
shard - Build an
MdbShardBuilder(plus, per xorb, its hash and concatenated raw bytes) from a chunk list, packing chunks into xorbs that respect the XETMAX_XORB_BYTES(64 MiB) andMAX_XORB_CHUNKS(8192) limits. - build_
shard_ with_ cap - Like
build_shard, but with a custom per-xorb byte cap (tests use a tiny cap to force multi-xorb aggregation without allocating 64 MiB).