A real-time voice agent rarely feels fast because of one model alone. It feels fast because speech recognition, reasoning, and speech synthesis start doing useful work before the previous stage has completely finished. That is the important...
A real-time voice agent rarely feels fast because of one model alone. It feels fast because speech recognition, reasoning, and speech synthesis start doing useful work before the previous stage has completely finished.
That is the important mental model when building with LiveKit.
LiveKit handles the real-time media and agent plumbing. Your STT and TTS choices still determine how quickly speech becomes usable text and how quickly generated text becomes audible speech. In this guide, we will connect LiveKit Agents with Smallest AI, using Pulse for speech-to-text and Lightning for text-to-speech.
The focus is not just getting a demo running. We will also look at endpointing, interruptions, failure recovery, state management, and the latency boundaries you should measure before shipping.
The pipeline is concurrent, not sequential
The basic architecture looks simple:
User audio -> STT -> LLM -> TTS -> User audio
But implementing those stages as a strictly sequential pipeline creates unnecessary waiting.
In a streaming voice system:
STT begins producing transcript updates while the user audio is still being processed.
The LLM can begin generating a response once enough text is available.
TTS can begin synthesizing useful text chunks before the full answer is complete.
Audio playback can begin while later text is still being generated and synthesized.
The stages overlap.
That is why first useful output matters so much in voice systems. Waiting for a complete transcript, then a complete LLM response, then a complete audio file makes each stage's latency accumulate.
LiveKit Agents is designed around this streaming model. Its plugin system also lets you replace individual STT, LLM, and TTS components without rewriting the complete media pipeline.
If you want the architecture argument in more detail, the guide on why streaming architecture matters for real-time voice agents explains why partial results and overlapping work matter so much for conversational latency.
If LiveKit Agents is new to you, the LiveKit Agents documentation is a useful starting point before continuing.
Set up the LiveKit and Smallest AI stack
For a current Python setup, you need:
A LiveKit Cloud project or your own LiveKit deployment
A Smallest AI account and API key
Credentials for the LLM provider you are using
Python 3.11 for the setup shown here
Create a virtual environment and install the required packages:
python3.11 -m venv .venv
source .venv/bin/activate
pip install livekit-plugins-smallestai livekit-plugins-openai livekit-plugins-silero python-dotenv
On Windows, activate the environment with the equivalent .venv\Scripts\activate command.
The important dependency here is livekit-plugins-smallestai. It exposes both the Pulse STT and Lightning TTS integrations, so you do not need separate provider packages for each side of the speech pipeline.
For parameters, supported models, and integration-specific configuration, keep Smallest AI's LiveKit integration alongside the LiveKit documentation while implementing.
Create and store the API key
The Smallest AI API uses authenticated access to the speech services used by this integration.
Keep the API key in an environment variable rather than hard-coding it into the application.
Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.
export SMALLEST_API_KEY="your-api-key-here"
Every authenticated request sends the value through the Authorization header:
Authorization: Bearer
Keep the key on your server. Do not expose it in browser JavaScript, mobile application code, public repositories, screenshots, query parameters, client-side logs, or error messages returned to users.
For production deployments, store it in your infrastructure's server-side secrets manager rather than committing it to source control.
You will also need your LiveKit credentials and whichever LLM-provider credentials your application uses. For the example below, that means configuring:
LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
OPENAI_API_KEY
SMALLEST_API_KEY
Wire Pulse STT and Lightning TTS into LiveKit
Current LiveKit Agents integrations use AgentSession, with each model supplied as a component of the session.
Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.
import os
from dotenv import load_dotenv
from livekit.agents import (
Agent,
AgentSession,
JobContext,
JobProcess,
WorkerOptions,
cli,
)
from livekit.plugins import openai, silero, smallestai
load_dotenv()
def require_env(name: str) -> str:
value = os.getenv(name)
if not value:
raise RuntimeError(f"{name} is not set")
return value
class VoiceAgent(Agent):
def __init__(self) -> None:
super().__init__(
instructions="You are a concise, helpful voice assistant."
)
def prewarm(proc: JobProcess) -> None:
proc.userdata["vad"] = silero.VAD.load()
async def entrypoint(ctx: JobContext) -> None:
smallest_api_key = require_env("SMALLEST_API_KEY")
require_env("LIVEKIT_URL")
require_env("LIVEKIT_API_KEY")
require_env("LIVEKIT_API_SECRET")
require_env("OPENAI_API_KEY")
session = AgentSession(
vad=ctx.proc.userdata["vad"],
stt=smallestai.STT(
api_key=smallest_api_key,
model="pulse",
language="en",
),
llm=openai.LLM(
model="gpt-4o-mini",
),
tts=smallestai.TTS(
api_key=smallest_api_key,
model="lightning_v3.1_pro",
voice_id="meher",
language="en",
),
)
await session.start(
agent=VoiceAgent(),
room=ctx.room,
)
if __name__ == "__main__":
cli.run_app(
WorkerOptions(
entrypoint_fnc=entrypoint,
prewarm_fnc=prewarm,
)
)
Run the agent in development mode:
python3 agent.py dev
The key architectural point is inside AgentSession.
smallestai.STT(...) handles the incoming speech layer. The LLM generates the response. smallestai.TTS(...) converts that response back into streaming speech.
LiveKit coordinates those stages as part of one session rather than requiring you to manually shuttle audio and text between independent services.
Model names, voice IDs, language options, and plugin behavior can change as releases evolve, so verify those values against the current documentation before pinning a production configuration.
What Pulse contributes to the pipeline
Pulse is the speech-to-text side of the integration.
For a real-time agent, the STT layer is not simply a function that returns a transcript. It participates in turn timing.
The application needs to answer questions such as:
When has the user actually finished speaking?
Do partial transcripts arrive early enough to help downstream processing?
How does background noise affect end-of-turn detection?
What happens when a user pauses in the middle of a sentence?
Are language settings correct for the traffic you actually receive?
LiveKit's STT integration can consume Pulse as a streaming provider, while the rest of the agent continues through the normal LiveKit session abstraction.
That separation is useful because you can evaluate speech recognition independently without redesigning your LLM and TTS layers.
Why the TTS stage affects perceived speed so much
A voice agent can have accurate transcription and a fast LLM and still feel slow.
The user does not hear either of those components directly. They hear the first audio produced by the TTS layer.
If the application waits for the entire LLM response before starting synthesis, you lose much of the benefit of streaming upstream.
Lightning fits the streaming TTS side of the LiveKit pipeline by turning generated text into audio as the interaction progresses rather than requiring a completed long-form response first.
This means your latency budget should not only measure total speech-generation time. Measure when the first useful audio reaches the listener.
For voice agents, that first audible response strongly affects whether the interaction feels continuous or delayed.
Think about each stage separately
A useful debugging model is to break the pipeline into boundaries:
Stage
Input
Output
What to watch
STT
User audio
Partial and final text
Turn timing, recognition delay, transcript stability
LLM
Conversation text
Generated response text
Time to first useful text, tool latency, context size
TTS
Generated text
Audio chunks
Time to first audio, buffering, cancellation behavior
Playback
Audio chunks
What the user hears
Queueing, interruption handling, transport delay
This makes performance problems easier to diagnose.
For example, a slow conversation does not automatically mean the TTS model is slow. The delay could come from endpointing, an LLM tool call, audio buffering, or the network path between your worker and a provider.
Measure boundaries independently before optimizing the wrong component.
Production concerns most demos do not expose
A successful demo proves that the pipeline is connected.
Production asks harder questions.
Endpointing and turn detection
The system must decide when the user's turn is complete.
Real users hesitate, restart sentences, speak over background noise, and pause before important words. An endpointing configuration that feels responsive for short test phrases can become frustrating with real conversation.
If your turn detector fires too early, the agent may cut the user off.
If it waits too long, every response begins with an awkward silence.
Recent LiveKit APIs are consolidating turn behavior under newer turn-handling configuration, so check the version-specific documentation instead of copying older min_endpointing_delay or max_endpointing_delay examples blindly.
There is another subtle issue: provider-side end-of-utterance logic and LiveKit turn detection can both contribute delay. Test the combined pipeline rather than tuning each component in isolation.
Barge-in and interruption handling
People interrupt each other naturally.
Your agent should expect the same behavior.
When the user starts speaking while the agent is talking, the system needs to stop or suppress the current playback, return attention to incoming speech, and continue the new turn without leaving stale audio queued behind.
LiveKit supports interruptible voice-agent flows, but you should still test the exact combinations your application will encounter:
Very short interruptions
Users correcting themselves
Interruptions during long TTS responses
Background speech that should not cancel playback
Repeated barge-in during a single session
Do not evaluate interruption handling with one happy-path sentence.
Network failures and provider errors
STT and TTS calls depend on network connections.
Transient errors will happen.
Your application should define what happens when:
The STT stream disconnects
TTS synthesis fails after text has already been generated
A request times out
The user disconnects during synthesis
A retry succeeds after the conversation has already moved on
Use bounded retries and exponential backoff where appropriate.
More importantly, make retries conversation-aware. Replaying a stale TTS response after the user has already started another turn is worse than dropping the response.
First-request connection overhead
Persistent streaming connections often behave differently on the first request than after a worker has warmed up.
If your deployment creates workers on demand, test cold-start behavior separately from steady-state performance.
For production voice AI, averages can hide a frustrating first interaction.
Multi-turn context is a separate design problem
Once latency is acceptable, long conversations introduce another issue: state.
A single-turn assistant can pass one transcript to the LLM and return one answer.
A multi-turn agent needs to decide how much previous conversation should remain in context.
Two common patterns are useful.
Sliding conversation window
Keep only the most recent N turns.
This is straightforward and works well when recent context matters more than information from much earlier in the session.
The tradeoff is obvious: once older turns fall out of the window, the model can no longer use them directly.
Periodic summarization
Instead of retaining every raw turn, periodically summarize older conversation state and keep that summary alongside the most recent exchanges.
This reduces context growth while preserving important information from earlier in the session.
The difficult part is deciding what the summary is allowed to discard.
For support, workflow, or transactional agents, preserving an account choice, requested action, or unresolved issue may matter much more than preserving exact wording.
Treat context management as application logic, not simply an LLM setting.
Measure the pipeline you actually plan to ship
A provider benchmark can tell you something useful about one model.
It cannot tell you the latency of your full application.
Measure at least these boundaries in your own deployment:
End of user speech to usable transcript
Usable transcript to first generated LLM text
First generated text to first TTS audio
End of user speech to first audible response
Time required to cancel playback after interruption
Error and reconnect rates during longer sessions
Run these tests with realistic network paths, audio devices, languages, sentence lengths, and concurrency.
A local development machine on fast Wi-Fi is not the production environment.
Frequently asked questions
Is Smallest AI officially supported by LiveKit?
Smallest AI appears in LiveKit's STT and TTS provider documentation, and the current integration uses the smallestai plugin package with AgentSession.
That is different from maintaining your own custom STT or TTS adapter outside LiveKit's plugin ecosystem.
What end-to-end latency should I expect?
There is no single number that applies to every deployment.
End-to-end latency includes turn detection, speech recognition, network transport, LLM generation, tool calls, speech synthesis, buffering, and playback.
Provider-level latency figures are useful for component evaluation, but measure the complete pipeline using your own deployment architecture.
Can I use a cloned voice?
Lightning supports voice-cloning workflows, but model and voice compatibility matters.
Verify that the Lightning model you choose supports the cloned voice you intend to use, then provide the corresponding voice_id to the TTS integration.
Do not assume that every voice is available across every Lightning model or language configuration.
Is LiveKit Agents free to use?
LiveKit Agents is open source.
Running an agent can still create infrastructure, bandwidth, model-provider, and other service costs depending on your deployment.
Which languages does Pulse support?
Pulse supports multilingual speech recognition, but supported languages and regional options can differ between real-time and pre-recorded transcription.
Check the current language list before deploying rather than hard-coding assumptions from an older model release.
Next steps
The most important lesson is not that one STT or TTS model makes an agent real-time.
The system feels conversational when the entire pipeline is designed to stream, overlap work, cancel obsolete work, and recover cleanly when real users behave unpredictably.
Start with the simple STT-LLM-TTS pipeline. Then measure each boundary independently. Tune endpointing and interruptions with real conversations. Finally, decide how your application will handle context, retries, cold starts, and long-running sessions.
When you are ready to test the speech layer with your own LiveKit application, start building with the Smallest AI API and measure the complete pipeline under the conditions your users will actually encounter.