Skip to content

rhapsody.telemetry.instrumentation.inference

Inference telemetry: batch spans, per-request queue waits and batch metrics.

The only place that knows the inference telemetry. The vLLM backend reports plain data through its emitter (two events per batch, rhapsody.inference.BatchStarted and BatchEnded); this instrumentation turns it into OpenTelemetry using the API only. It records:

  • one chat {model} span per batch (SpanKind.CLIENT) with the batch's real start and end times, parented to the session span and linked to the task spans of the batch;
  • the per-task facts (model, batch id, prompt count, queue wait, timeouts) as attributes of the open task spans;
  • histograms of the request queue wait, the batch duration and the Dragon model duration, and counters of requests, prompts, timed-out prompts and batches.

No prompt or response text is ever carried. The feature is experimental: the GenAI semantic conventions it follows are still in development.

InferenceInstrumentation 

InferenceInstrumentation()

Bases: Instrumentation

Records the telemetry of the vLLM batcher from its rhapsody.inference.* events.

Source code in src/rhapsody/telemetry/instrumentation/inference.py 
65
66
67
68
69
def __init__(self) -> None:
    self._handlers = {
        _PREFIX + "BatchStarted": self._on_batch_started,
        _PREFIX + "BatchEnded": self._on_batch_ended,
    }