Term tag: latency

Model Routing

Model routing sends each request to the cheapest capable model — a small model handles the easy majority and hard queries escalate to a frontier model.

Performance

AI performance is not one number but a frontier of quality, speed, and cost—improving one axis (via quantization or batching) usually trades off another.

Production Systems

A model serving live users under latency and uptime guarantees — judged at the 99th percentile under real load, not by the median on a laptop.

Speech-to-Speech

Speech-to-speech AI takes audio in and returns audio out — conversation or translation — via a cascade (ASR to text to TTS) or one end-to-end model.