Struct XetStorage
pub struct XetStorage { /* private fields */ }Expand description
XET storage backend implementation.
Wraps the XET data client to provide clean/smudge operations for git filter integration. Uses async I/O for all network operations.
§Example
use git_xet_filter::storage::XetStorage;
use git_xet_filter::config::XetConfig;
let config = XetConfig::huggingface();
let storage = XetStorage::new(&config).await?;
// Clean (upload) a file
let pointer = storage.clean_file(Path::new("weights.safetensors")).await?;
// Smudge (download) to memory
let data = storage.smudge_bytes(&pointer).await?;Implementations§
§impl XetStorage
impl XetStorage
pub async fn new(config: &XetConfig) -> Result<XetStorage, XetError>
pub async fn new(config: &XetConfig) -> Result<XetStorage, XetError>
Create a new XET storage backend with the given configuration.
Initializes the upload session and downloader with the endpoint and authentication from the config.
§Errors
Returns an error if the XET data client fails to initialize, typically due to network issues or invalid credentials.
pub async fn clean_bytes(&self, data: &[u8]) -> Result<String, XetError>
pub async fn clean_bytes(&self, data: &[u8]) -> Result<String, XetError>
Upload data from memory and return XET pointer JSON
This is more efficient than writing to a temp file first when the data is already in memory (e.g., from tensor serialization).
pub async fn smudge_file(
&self,
pointer: &str,
output_path: &Path,
) -> Result<(), XetError>
pub async fn smudge_file( &self, pointer: &str, output_path: &Path, ) -> Result<(), XetError>
Download from XET pointer directly to a file path
This is more efficient than downloading to memory first when the final destination is a file (e.g., model weights).
pub async fn smudge_file_bounded(
&self,
pointer: &str,
output_path: &Path,
max_output_bytes: u64,
) -> Result<u64, XetError>
pub async fn smudge_file_bounded( &self, pointer: &str, output_path: &Path, max_output_bytes: u64, ) -> Result<u64, XetError>
Download a pointer into a new file while enforcing max_output_bytes
at each sequential write. This is intended for callers that must bound
untrusted or stale pointer metadata before the payload reaches disk.
pub async fn smudge_hash_to_file_bounded(
&self,
hash: &DataHash,
output_path: &Path,
max_output_bytes: u64,
) -> Result<u64, XetError>
pub async fn smudge_hash_to_file_bounded( &self, hash: &DataHash, output_path: &Path, max_output_bytes: u64, ) -> Result<u64, XetError>
Download content by Merkle hash using a strict write-time byte ceiling.
pub async fn smudge_bytes(&self, pointer: &str) -> Result<Vec<u8>, XetError>
pub async fn smudge_bytes(&self, pointer: &str) -> Result<Vec<u8>, XetError>
Download from XET pointer to memory
This is a better-named version of smudge_pointer() and uses async I/O.
pub async fn smudge_from_hash(
&self,
hash: &DataHash,
) -> Result<Vec<u8>, XetError>
pub async fn smudge_from_hash( &self, hash: &DataHash, ) -> Result<Vec<u8>, XetError>
Download from XET by merkle hash to memory
This is useful for LFS pointer resolution where you have a SHA256 hash and need to retrieve the content directly without a XET pointer.
pub async fn smudge_from_hash_to_file(
&self,
hash: &DataHash,
output_path: &Path,
) -> Result<(), XetError>
pub async fn smudge_from_hash_to_file( &self, hash: &DataHash, output_path: &Path, ) -> Result<(), XetError>
Download from XET by merkle hash directly to a file path
This is useful for LFS pointer resolution where you have a SHA256 hash and want to write directly to disk.
Trait Implementations§
Source§impl LfsSmudge for XetStorage
impl LfsSmudge for XetStorage
Source§fn smudge_lfs<'life0, 'life1, 'async_trait>(
&'life0 self,
pointer: &'life1 LfsPointer,
) -> Pin<Box<dyn Future<Output = Git2DBResult<Vec<u8>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn smudge_lfs<'life0, 'life1, 'async_trait>(
&'life0 self,
pointer: &'life1 LfsPointer,
) -> Pin<Box<dyn Future<Output = Git2DBResult<Vec<u8>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Source§fn smudge_lfs_to_file<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
pointer: &'life1 LfsPointer,
path: &'life2 Path,
) -> Pin<Box<dyn Future<Output = Git2DBResult<()>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn smudge_lfs_to_file<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
pointer: &'life1 LfsPointer,
path: &'life2 Path,
) -> Pin<Box<dyn Future<Output = Git2DBResult<()>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
Source§fn smudge_lfs_validated<'life0, 'life1, 'async_trait>(
&'life0 self,
pointer: &'life1 LfsPointer,
) -> Pin<Box<dyn Future<Output = Git2DBResult<Vec<u8>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn smudge_lfs_validated<'life0, 'life1, 'async_trait>(
&'life0 self,
pointer: &'life1 LfsPointer,
) -> Pin<Box<dyn Future<Output = Git2DBResult<Vec<u8>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
§impl StorageBackend for XetStorage
impl StorageBackend for XetStorage
§fn clean_file<'life0, 'life1, 'async_trait>(
&'life0 self,
path: &'life1 Path,
) -> Pin<Box<dyn Future<Output = Result<String, XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
XetStorage: 'async_trait,
fn clean_file<'life0, 'life1, 'async_trait>(
&'life0 self,
path: &'life1 Path,
) -> Pin<Box<dyn Future<Output = Result<String, XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
XetStorage: 'async_trait,
§fn is_pointer(&self, content: &str) -> bool
fn is_pointer(&self, content: &str) -> bool
§fn clean_bytes<'life0, 'life1, 'async_trait>(
&'life0 self,
data: &'life1 [u8],
) -> Pin<Box<dyn Future<Output = Result<String, XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
XetStorage: 'async_trait,
fn clean_bytes<'life0, 'life1, 'async_trait>(
&'life0 self,
data: &'life1 [u8],
) -> Pin<Box<dyn Future<Output = Result<String, XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
XetStorage: 'async_trait,
§fn smudge_file<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
pointer: &'life1 str,
output_path: &'life2 Path,
) -> Pin<Box<dyn Future<Output = Result<(), XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
XetStorage: 'async_trait,
fn smudge_file<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
pointer: &'life1 str,
output_path: &'life2 Path,
) -> Pin<Box<dyn Future<Output = Result<(), XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
XetStorage: 'async_trait,
§fn smudge_bytes<'life0, 'life1, 'async_trait>(
&'life0 self,
pointer: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Vec<u8>, XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
XetStorage: 'async_trait,
fn smudge_bytes<'life0, 'life1, 'async_trait>(
&'life0 self,
pointer: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Vec<u8>, XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
XetStorage: 'async_trait,
§fn smudge_from_hash<'life0, 'life1, 'async_trait>(
&'life0 self,
hash: &'life1 DataHash,
) -> Pin<Box<dyn Future<Output = Result<Vec<u8>, XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
XetStorage: 'async_trait,
fn smudge_from_hash<'life0, 'life1, 'async_trait>(
&'life0 self,
hash: &'life1 DataHash,
) -> Pin<Box<dyn Future<Output = Result<Vec<u8>, XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
XetStorage: 'async_trait,
§fn smudge_from_hash_to_file<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
hash: &'life1 DataHash,
output_path: &'life2 Path,
) -> Pin<Box<dyn Future<Output = Result<(), XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
XetStorage: 'async_trait,
fn smudge_from_hash_to_file<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
hash: &'life1 DataHash,
output_path: &'life2 Path,
) -> Pin<Box<dyn Future<Output = Result<(), XetError>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
XetStorage: 'async_trait,
Auto Trait Implementations§
impl Freeze for XetStorage
impl !RefUnwindSafe for XetStorage
impl Send for XetStorage
impl Sync for XetStorage
impl Unpin for XetStorage
impl UnsafeUnpin for XetStorage
impl !UnwindSafe for XetStorage
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more