Skip to main content

Module token_metrics

Module token_metrics 

Source
Expand description

OpenTelemetry token-burn metering for inference.

Token burn — prompt tokens consumed plus generated tokens produced per request — is accounting- and ops-relevant, so it is emitted as OpenTelemetry metrics rather than process logs (#1253, #1261). The authoritative spend ledger lives in #1264; the instruments here are the metering/ops signal.

§Instruments

  • inference_tokens_total — Counter<u64> (unit {token}, UCUM custom-unit form), cumulative tokens consumed, with attributes kind = prompt | generated, model, and an opaque keyed-hash tenant id.
  • inference_request_tokens — Histogram<u64> (unit {token}), total tokens (prompt + generated) per request, with model / tenant attributes only.
  • inference_speculative_tokens_total — Counter<u64> (unit {token}), MTP self-speculative draft outcomes, with attribute kind = accepted | rejected plus model / opaque tenant. Integer draft counts only — never token IDs or text.

§Security

The raw request subject is never used as a metric attribute: metrics are aggregatable across tenants and must not carry tenant-identifying content. The tenant is reduced to an opaque keyed hash via opaque_tenant_id before it becomes an attribute. Prompt text, token text, generated text, and token IDs are likewise never in metrics — only integer counts. See the no-leak canary test in crate::runtime::torch_engine which guards the same property for logs.

§Platform gating

The opentelemetry dependency family is not wasm32-compatible, so the crates are target-gated in Cargo.toml and the instruments here compile only for all(feature = "otel", not(target_arch = "wasm32")). Everywhere else a no-op TokenBurnMeter keeps the call sites compiling — the wasm/browser client pays no metric cost and non-otel builds meter nothing.

Structs§

TokenBurnMeter
Token-burn instruments built from an OTel [Meter].

Functions§

global_meter
Shared process-global token-burn instruments.
opaque_tenant_id
Reduce a raw tenant subject to an opaque 64-bit id for metric attrs and tenant-safe logs.