Skip to main content

XetStorage

Struct XetStorage 

Source
pub struct XetStorage { /* private fields */ }
Expand description

XET storage backend implementation.

Wraps the XET data client to provide clean/smudge operations for git filter integration. Uses async I/O for all network operations.

§Example

ⓘ
use git_xet_filter::storage::XetStorage;
use git_xet_filter::config::XetConfig;

let config = XetConfig::huggingface();
let storage = XetStorage::new(&config).await?;

// Clean (upload) a file
let pointer = storage.clean_file(Path::new("weights.safetensors")).await?;

// Smudge (download) to memory
let data = storage.smudge_bytes(&pointer).await?;

Implementations§

Source§

impl XetStorage

Source

pub async fn new(config: &XetConfig) -> Result<Self>

Create a new XET storage backend with the given configuration.

Initializes the upload session and downloader with the endpoint and authentication from the config.

§Errors

Returns an error if the XET data client fails to initialize, typically due to network issues or invalid credentials.

Source

pub async fn clean_bytes(&self, data: &[u8]) -> Result<String>

Upload data from memory and return XET pointer JSON

This is more efficient than writing to a temp file first when the data is already in memory (e.g., from tensor serialization).

Source

pub async fn smudge_file(&self, pointer: &str, output_path: &Path) -> Result<()>

Download from XET pointer directly to a file path

This is more efficient than downloading to memory first when the final destination is a file (e.g., model weights).

Source

pub async fn smudge_file_bounded( &self, pointer: &str, output_path: &Path, max_output_bytes: u64, ) -> Result<u64>

Download a pointer into a new file while enforcing max_output_bytes at each sequential write. This is intended for callers that must bound untrusted or stale pointer metadata before the payload reaches disk.

Source

pub async fn smudge_hash_to_file_bounded( &self, hash: &MerkleHash, output_path: &Path, max_output_bytes: u64, ) -> Result<u64>

Download content by Merkle hash using a strict write-time byte ceiling.

Source

pub async fn smudge_bytes(&self, pointer: &str) -> Result<Vec<u8>>

Download from XET pointer to memory

This is a better-named version of smudge_pointer() and uses async I/O.

Source

pub async fn smudge_from_hash(&self, hash: &MerkleHash) -> Result<Vec<u8>>

Download from XET by merkle hash to memory

This is useful for LFS pointer resolution where you have a SHA256 hash and need to retrieve the content directly without a XET pointer.

Source

pub async fn smudge_from_hash_to_file( &self, hash: &MerkleHash, output_path: &Path, ) -> Result<()>

Download from XET by merkle hash directly to a file path

This is useful for LFS pointer resolution where you have a SHA256 hash and want to write directly to disk.

Trait Implementations§

Source§

impl StorageBackend for XetStorage

Source§

fn clean_file<'life0, 'life1, 'async_trait>( &'life0 self, path: &'life1 Path, ) -> Pin<Box<dyn Future<Output = Result<String>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Upload a file to XET CAS and return a JSON pointer string. Read more
Source§

fn is_pointer(&self, content: &str) -> bool

Check if a string is a valid XET pointer. Read more
Source§

fn clean_bytes<'life0, 'life1, 'async_trait>( &'life0 self, data: &'life1 [u8], ) -> Pin<Box<dyn Future<Output = Result<String>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Upload data from memory to XET CAS and return a JSON pointer string. Read more
Source§

fn smudge_file<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, pointer: &'life1 str, output_path: &'life2 Path, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Download content from a pointer directly to a file. Read more
Source§

fn smudge_bytes<'life0, 'life1, 'async_trait>( &'life0 self, pointer: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<Vec<u8>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Download content from a pointer to an in-memory buffer. Read more
Source§

fn smudge_from_hash<'life0, 'life1, 'async_trait>( &'life0 self, hash: &'life1 MerkleHash, ) -> Pin<Box<dyn Future<Output = Result<Vec<u8>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Download content by merkle hash to an in-memory buffer. Read more
Source§

fn smudge_from_hash_to_file<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, hash: &'life1 MerkleHash, output_path: &'life2 Path, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Download content by merkle hash directly to a file. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
§

impl<T> Any for T
where T: Any,

§

fn into_any(self: Box<T>) -> Box<dyn Any>

§

fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>

§

fn type_name(&self) -> &'static str

§

impl<T> AnySync for T
where T: Any + Send + Sync,

§

fn into_any_arc(self: Arc<T>) -> Arc<dyn Any + Sync + Send> ⓘ

Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T, W> HasTypeWitness<W> for T
where W: MakeTypeWitness<Arg = T>, T: ?Sized,

§

const WITNESS: W = W::MAKE

A constant of the type witness
§

impl<T> Identity for T
where T: ?Sized,

§

const TYPE_EQ: TypeEq<T, <T as Identity>::Type> = TypeEq::NEW

Proof that Self is the same type as Self::Type, provides methods for casting between Self and Self::Type.
§

type Type = T

The same type as Self, used to emulate type equality bounds (T == U) with associated type equality constraints (T: Identity<Type = U>).
§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self>

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
§

impl<T> Pointable for T

§

const ALIGN: usize

The alignment of pointer.
§

type Init = T

The type for initializers.
§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
§

impl<T> ServiceExt for T

§

fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>
where Self: Sized,

Apply a transformation to the response body. Read more
§

fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using HTTP status codes. Read more
§

fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using gRPC headers. Read more
§

fn follow_redirects(self) -> FollowRedirect<Self>
where Self: Sized,

Follow redirect resposes using the Standard policy. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

impl<T> ErasedDestructor for T
where T: 'static,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSync for T
where T: Sync,

§

impl<T> MaybeSync for T
where T: Sync,