Skip to main content

Module shard

Module shard 

Source
Expand description

Reconstruction shard — the XET mdb_shard binary reconstruction format.

A shard (the .mdb file) carries the File-Info + CAS-Info sections that map a file to the ordered sequence of (xorb hash, chunk-index range, byte length) segments needed to reassemble it. This module is the cas-serve integration layer over crate::mdb_shard: it owns the chunk → xorb → shard pipeline and exposes the helpers the binary’s request handlers use.

The shard is content-addressed by the file’s merkle hash and stored at shards/{file_hash} as the raw .mdb bytes. Reconstruction (GetFile) loads the shard, walks its File-Info segments, fetches each referenced xorb, slices out the chunk range, and concatenates the bytes in order.

This is the critical-path BYTES layer for multi-xorb (i.e. >64 MiB) file transfer described in #390. Unlike the earlier bespoke JSON shard, this format is byte-compatible with upstream xet-core (MDBShardFile / FileReconstructor), so a stock xet-core client can consume the shard returned by GetReconstructionInfo.

Structs§

Segment
One reconstruction segment: the xorb that holds the bytes, the chunk-index range within that xorb, and the total byte length of those chunks. This is a deserialized view of one mdb_shard FileDataSequenceEntry, exposed for callers (e.g. GetFile) that reassemble without keeping a full xorb chunk table in memory.
Shard
The reconstruction manifest for a single file: the .mdb bytes plus the derived metadata the protocol response carries (file hash, length, xorb hashes).