Stream everywhere, function calling for truth, barge-in, compliant opt-out, human handover: the six patterns that turn an LLM voice agent into a production tool.
Wiring an LLM agent to a phone line via Vapi + SIP (IllyVoIP, Vonage) is technically appealing. In production, it's a minefield. Field notes.
On an agency mission, the client wanted to automate outbound qualification calls (B2B, appointment booking, follow-up). Specs: natural-feeling voice agent, CRM integration, human operators taking over on demand, legal compliance (consent, opt-out).
Final stack: Vapi for voice orchestration + SIP (IllyVoIP backup, Vonage primary) for PSTN connectivity, OpenAI GPT-4 for conversational logic, N8N for the CRM pipeline, PostgreSQL for logs.
The goal of this article: share what worked, what blew up in flight, and how I'd restart today.
The LLM voice agent ticks every hype box: automation, generative AI, scale, marketing ROI. But under the hood, it's the integration of five asynchronous systems talking in real time:
At 3 seconds cumulative latency, the agent feels "slow and lost". At 1.2 seconds, "robot but OK". Under 800 ms, "almost human". Latency optimization is the project, not a detail.
| 1 | [User] ← PSTN/SIP → [IllyVoIP / Vonage] |
| 2 | │ |
| 3 | ▼ |
| 4 | [Vapi] |
| 5 | ╱ │ ╲ |
| 6 | [Deepgram] │ [ElevenLabs] |
| 7 | STT │ TTS |
| 8 | ▼ |
| 9 | [Agent API] |
| 10 | Node.js |
| 11 | │ |
| 12 | ┌──────────┼──────────┐ |
| 13 | ▼ ▼ ▼ |
| 14 | [OpenAI] [Postgres] [N8N] |
| 15 | reasoning logs CRM |
Vapi orchestrates the audio loop (STT → agent → TTS) with bidirectional streaming. Agent API (my code) receives transcribed text, calls OpenAI, returns the response to speak. Postgres traces everything. N8N integrates with the CRM.
Absolute rule: never wait for a step to finish before starting the next.
| 1 | // src/agent/handleTurn.ts |
| 2 | export async function handleTurn(userText: string, context: TurnContext) { |
| 3 | const stream = await openai.chat.completions.create({ |
| 4 | model: "gpt-4o", |
| 5 | messages: [...context.history, { role: "user", content: userText }], |
| 6 | stream: true, |
| 7 | temperature: 0.4, |
| 8 | }); |
| 9 | |
| 10 | let buffer = ""; |
| 11 | for await (const chunk of stream) { |
| 12 | const delta = chunk.choices[0]?.delta?.content ?? ""; |
| 13 | buffer += delta; |
| 14 | |
| 15 | const sentenceEnd = buffer.search(/[.!?]\s/); |
| 16 | if (sentenceEnd > 0) { |
| 17 | const sentence = buffer.slice(0, sentenceEnd + 1); |
| 18 | buffer = buffer.slice(sentenceEnd + 2); |
| 19 | await vapi.speak(sentence); |
| 20 | } |
| 21 | } |
| 22 | |
| 23 | if (buffer.trim()) await vapi.speak(buffer); |
| 24 | } |
Measured benefit: perceived latency divided by 3. Instead of waiting 2 seconds of generation then 1 second of TTS, the user hears the first sentence after 700 ms.
A voice agent isn't a text chatbot. Three specific requirements:
| 1 | You are Sarah, a sales assistant at Acme Corp. |
| 2 | |
| 3 | Mandatory rules: |
| 4 | - Short sentences (max 15 words) |
| 5 | - One question per turn, never more |
| 6 | - Echo aloud names, emails, dates and numbers for confirmation |
| 7 | - If you didn't catch something, say "I didn't catch that, could you repeat?" |
| 8 | - Never invent information |
| 9 | |
| 10 | Call objective: |
| 11 | - Qualify the prospect (budget, need, deadline) |
| 12 | - Propose an appointment within next week |
| 13 | - Capture email and best slot |
| 14 | |
| 15 | If the prospect refuses: thank politely and offer a definitive opt-out. |
Rule order matters. GPT-4 and Claude apply constraints stated first more reliably.
The user will interrupt the agent. Normal and healthy. The agent must stop immediately, not keep talking 3 seconds over the user.
Vapi exposes a barge-in event when STT detects the user starts speaking. Agent side:
| 1 | vapi.on("barge-in", () => { |
| 2 | vapi.stopSpeaking(); |
| 3 | currentTurn?.abort(); |
| 4 | vapi.resumeListening(); |
| 5 | }); |
Without this, the agent feels "deaf" and the experience becomes frustrating. It's the detail that moves "weird robot" to "acceptable agent".
The agent must NEVER invent a phone number, an email, an availability. Anything touching reality goes through structured function calling.
| 1 | const tools = [ |
| 2 | { |
| 3 | type: "function", |
| 4 | function: { |
| 5 | name: "checkAvailability", |
| 6 | description: "Check available slots in the calendar", |
| 7 | parameters: { |
| 8 | type: "object", |
| 9 | properties: { |
| 10 | startDate: { type: "string", format: "date" }, |
| 11 | endDate: { type: "string", format: "date" }, |
| 12 | durationMin: { type: "number" }, |
| 13 | }, |
| 14 | required: ["startDate", "endDate", "durationMin"], |
| 15 | }, |
| 16 | }, |
| 17 | }, |
| 18 | { |
| 19 | type: "function", |
| 20 | function: { |
| 21 | name: "bookAppointment", |
| 22 | description: "Book an appointment (call only after user confirmation)", |
| 23 | parameters: { |
| 24 | type: "object", |
| 25 | properties: { |
| 26 | email: { type: "string", format: "email" }, |
| 27 | name: { type: "string" }, |
| 28 | slotIso: { type: "string", format: "date-time" }, |
| 29 | }, |
| 30 | required: ["email", "name", "slotIso"], |
| 31 | }, |
| 32 | }, |
| 33 | }, |
| 34 | ]; |
When GPT-4 decides to call checkAvailability, my code actually queries the sales rep's Google Calendar, returns slots, GPT-4 phrases the answer. No hallucination possible.
Important: bookAppointment callable only after explicit confirmation ("I confirm your appointment Tuesday 2 PM, correct?" → "yes"). The system prompt enforces this discipline.
Three legal requirements in France/EU for automated outbound calls:
| 1 | const OPT_OUT_PHRASES = [ |
| 2 | "don't call me again", |
| 3 | "stop contacting me", |
| 4 | "remove me", |
| 5 | "gdpr", |
| 6 | "opt out", |
| 7 | ]; |
| 8 | |
| 9 | export function detectOptOut(userText: string): boolean { |
| 10 | const normalized = userText.toLowerCase().normalize("NFD").replace(/[\u0300-\u036f]/g, ""); |
| 11 | return OPT_OUT_PHRASES.some((phrase) => normalized.includes(phrase)); |
| 12 | } |
| 13 | |
| 14 | if (detectOptOut(userText)) { |
| 15 | await vapi.speak("Understood, I'm noting your request. You won't receive calls from us anymore. Have a good day."); |
| 16 | await OptOut.create({ phone: context.phone, reason: "user_request", at: new Date() }); |
| 17 | await vapi.endCall(); |
| 18 | return; |
| 19 | } |
Opt-out is non-negotiable. No "are you sure?", no "can I offer something else?". Close, log, respect.
No LLM agent replaces a human for sensitive cases. Detect when to transfer:
| 1 | async function transferToHuman(reason: string, context: TurnContext) { |
| 2 | await vapi.speak("I'll connect you with a counselor, please hold."); |
| 3 | await Transfer.create({ callId: context.callId, reason, at: new Date() }); |
| 4 | await vapi.transferCall({ destination: HUMAN_QUEUE_NUMBER }); |
| 5 | } |
The agent must know "I can't do this" without shame, and hand off fast.
Trace each call on three axes:
| Metric | Definition | Acceptable threshold |
|---|---|---|
| p95 latency per turn | User-stops-speaking → agent-starts | < 1200 ms |
| Barge-in rate | % calls with ≥1 interrupt | Stable around 20-30% |
| Opt-out rate | % calls ending in opt-out | < 5% |
| Handover rate | % calls transferred to human | < 15% |
| Completion rate | % calls reaching objective | > 30% |
If latency drifts, experience explodes in two days. Monitor continuously.
| Pitfall | Symptom | Fix |
|---|---|---|
| TTS without streaming | Unbearable perceived latency | Sentence-by-sentence streaming |
| No barge-in | Agent talks over user | barge-in listener + stopSpeaking |
| Date/email hallucination | Wrong appointment booked, fake email | Function calling for all real facts |
| Fuzzy opt-out | Legal risk | Lexical detection + log + end call |
| No handover | Guaranteed frustration | Multi-signal detection + transfer |
| Over-long system prompt | Model hallucinates more | Max 800 tokens, ordered rules |
| No do-not-call check | Legal risk | Systematic check before each call |
Measured cost per call (B2B qualification, 3 min average):
Vs a human sales rep: ~€3 to €5 per call (loaded salary + overhead). Real savings, but assume substantial call volume — under 200 calls/day, human stays better on relationship quality.
The LLM voice agent isn't a gadget. It's a professional tool when you respect three principles: stream everywhere, function calling for truth, legal compliance by default. Without these three, it's a demo that doesn't hold in production.
Nor is the agent a universal replacement for human sales. It excels at massive standardized qualification. It fails on nuance, negotiation and emotion. Hybrid human + agent stays the best economic calculation.
If starting: Vapi + GPT-4o + Deepgram + ElevenLabs is the stack that worked best for me in 2026. But count 6 to 10 weeks to ship a truly production-ready agent, not the weekend sprint sold on LinkedIn.
If this topic feels close to a real product problem, I can help on diagnosis, architecture, backend, interface and automations that make a platform usable in production.
Reader reactions
No comment yet
Be the first to share your reaction.