pub struct TensorSampler { /* private fields */ }Expand description
Device-agnostic token sampler operating on tensors
Performs all sampling operations (temperature scaling, top-k filtering, top-p nucleus sampling) directly on tensors, allowing the PyTorch backend to optimize for the target device (CPU, CUDA, ROCm, Metal, etc.).
Implementations§
Source§impl TensorSampler
impl TensorSampler
pub fn new(device: Device) -> Self
Sourcepub fn set_seed(seed: u64)
pub fn set_seed(seed: u64)
Set the random seed for deterministic sampling This enables reproducible token generation for debugging
Sourcepub fn sample_token(
&self,
logits_tensor: &Tensor,
temperature: f32,
top_p: f32,
top_k: Option<usize>,
repeat_penalty: f32,
previous_tokens: &[i64],
) -> Result<usize>
pub fn sample_token( &self, logits_tensor: &Tensor, temperature: f32, top_p: f32, top_k: Option<usize>, repeat_penalty: f32, previous_tokens: &[i64], ) -> Result<usize>
Sample next token directly from logits tensor
Sourcepub fn sample_token_with_penalty_exemptions(
&self,
logits_tensor: &Tensor,
temperature: f32,
top_p: f32,
top_k: Option<usize>,
repeat_penalty: f32,
previous_tokens: &[i64],
exempt_tokens: &HashSet<i64>,
) -> Result<usize>
pub fn sample_token_with_penalty_exemptions( &self, logits_tensor: &Tensor, temperature: f32, top_p: f32, top_k: Option<usize>, repeat_penalty: f32, previous_tokens: &[i64], exempt_tokens: &HashSet<i64>, ) -> Result<usize>
Sample a token with reduced repeat penalty for single-character tokens.
Exempt tokens (digits, punctuation) receive sqrt(penalty) instead of
the full penalty, preventing digit suppression in year generation while
still discouraging runaway single-character repetition.
Trait Implementations§
Source§impl Clone for TensorSampler
impl Clone for TensorSampler
Source§fn clone(&self) -> TensorSampler
fn clone(&self) -> TensorSampler
Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source. Read moreAuto Trait Implementations§
impl Freeze for TensorSampler
impl RefUnwindSafe for TensorSampler
impl Send for TensorSampler
impl Sync for TensorSampler
impl Unpin for TensorSampler
impl UnsafeUnpin for TensorSampler
impl UnwindSafe for TensorSampler
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more§impl<T> Pointable for T
impl<T> Pointable for T
§impl<T> PolicyExt for Twhere
T: ?Sized,
impl<T> PolicyExt for Twhere
T: ?Sized,
§impl<T> ServiceExt for T
impl<T> ServiceExt for T
§fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>where
Self: Sized,
fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>where
Self: Sized,
Apply a transformation to the response body. Read more
§fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>where
Self: Sized,
fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>where
Self: Sized,
High level tracing that classifies responses using HTTP status codes. Read more
§fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>where
Self: Sized,
fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>where
Self: Sized,
High level tracing that classifies responses using gRPC headers. Read more