How to build a Customer Support Voice Agent

Voice AI agent demo often looks simple:

Caller → Speech-to-Text → LLM → Text-to-Speech → Caller

In production systems this looks very different.

Between those components lies the real engineering involved in low-latency audio streaming, turn detection, interruption handling, telephony codecs, webhook orchestration, and voicemail detection.

In this tutorial, we'll build a customer support voice agent without assembling that pipeline yourself. Using Telnyx AI Assistant Builder, you'll configure the entire voice stack from a portal and write a small FastAPI webhook that injects live business context into every call.

[

](https://x.com/Sumanth_077/article/2076671269391696042/media/2076668833600081920)

What a working customer support voice agent actually requires

The same core building blocks, regardless of the tools you choose:

Sub-second round-trip latency. STT, the language model, TTS, and the telephony layer all have to fit inside about one second, or the conversation feels awkward.
Grounded answers. The agent has to speak from a knowledge base, not from vibes. Hallucinated policies hurt more than they help.
Live context per call. Queue wait, incidents in the last hour, whether the branch is open right now. None of that can live in a static prompt.
Graceful handoff. Some questions do not get answered by AI. The agent needs to know when to stop trying and pass the caller to a human, cleanly.
Testable without a phone number. You should be able to iterate on the agent's instructions without dialling a number and burning a call minute every time.

This article covers all of them. Instead of assembling providers, you configure one platform in a portal and write code only for the context the assistant needs when a call begins.

The Telnyx AI Assistant Builder

Telnyx's AI Assistant Builder

is a

portal

for creating a voice assistant. You give it a name, pick a language model, pick a voice, pick a transcription provider, fill in the Instructions field (the system prompt that carries the assistant's persona, response format, and knowledge base), and attach a phone number. The platform handles the rest:

STT (Telnyx native, Deepgram, or Azure)
The language model (Kimi K2.5 on Telnyx infrastructure with no API key, GPT-4o via OpenAI, and several others)
TTS (Telnyx native, ElevenLabs, AWS, Azure, Inworld)
The phone number and PSTN routing
Conversation history and playback
An embeddable browser widget that connects to the same assistant over WebRTC

[

](https://x.com/Sumanth_077/article/2076671269391696042/media/2076669031504076801)

The Instructions field is where the assistant's persona lives. For the tutorial this article follows, the persona is a fictional retail bank:

You are a customer support agent for a full-service retail bank serving personal and business customers. Keep all responses under 2 sentences -- callers are listening, not reading. Speak calmly and professionally. If a question is not covered in the knowledge base below, acknowledge it warmly and offer to transfer the caller to a specialist. System status: {{system_status}} Active incidents: {{todays_incidents}} Queue wait: {{current_queue_wait}} Business hours: {{business_hours}} Support email: {{support_email}} --- KNOWLEDGE BASE --- ACCOUNTS We offer three personal account tiers. The Everyday Checking account carries no minimum balance, includes a free debit card and online bill pay, and reimburses up to $10 in ATM fees each month...

Dynamic Variables

The Instructions field holds information that stays the same across every call. Information that changes over time, such as the current queue wait time, active incidents, or whether a branch is open, is handled by Dynamic Variables.

Anywhere in the Instructions field you write {{variable_name}}, Telnyx replaces it with a value when the call begins. Those values come from a webhook you host.

The flow is simple:

Caller dials the number
Telnyx sends an HTTP POST to your webhook URL
Your webhook returns a JSON object of key-value pairs
Telnyx substitutes them into the system prompt
The model reads the substituted prompt for the rest of the call

For a support agent, every call starts with fresh values for queue wait times, active incidents, business hours, and anything else you want to keep current.

The webhook is just a single FastAPI route:

python

from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse
import telnyx, os

telnyx_client = telnyx.Telnyx(
    api_key=os.getenv("TELNYX_API_KEY"),
    public_key=os.getenv("TELNYX_PUBLIC_KEY"),
)

app = FastAPI()

def get_live_context() -> dict:
    return {
        "system_status": "All systems operational",
        "current_queue_wait": "Under 2 minutes",
        "business_hours": "Monday to Friday, 9AM to 5PM EST; Saturday, 9AM to 1PM EST",
        "todays_incidents": "None",
        "support_email": "support@bank.com",
    }

@app.post("/webhooks/dynamic-variables")
async def dynamic_variables(request: Request) -> JSONResponse:
    raw_body = await request.body()
    try:
        telnyx_client.webhooks.unwrap(
            raw_body.decode("utf-8"), dict(request.headers)
        )
    except Exception:
        raise HTTPException(status_code=401, detail="Invalid webhook signature")

    return JSONResponse(get_live_context())

That is the whole webhook. Roughly fifteen effective lines of code. No STT integration. No language model API key. No TTS work. No telephony. Just the piece that has to change per call.

Full Workflow

Here is the full flow, from dialed number to answered question:

Caller dials the Telnyx phone number. Telnyx routes the call to the AI Assistant configured for that number.
Telnyx fires the Dynamic Variables webhook. POST to your FastAPI endpoint with call metadata in the body.
The webhook verifies the signature and returns a context dict. Live queue wait, incidents, business hours.
Telnyx substitutes those values into the Instructions field. The {{variable}} placeholders become their current values.
The model reads the fully-substituted prompt. Persona, knowledge base, and live context in one system message.
STT, language model, and TTS run on Telnyx infrastructure. No hops to external providers. The audio stays inside one platform.
The conversation happens. Caller asks about card fraud, wire transfer times, or the current queue wait. The model answers from the KB and the live context.

[

](https://x.com/Sumanth_077/article/2076671269391696042/media/2076668510621802496)

Getting the agent running

Clone the repo:

bash

git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git
cd Hands-On-AI-Engineering/voice_apps/customer_support_voice_agent
uv venv
source .venv/bin/activate
uv pip install -e .

Set up credentials:

bash

cp .env.example .env
# Edit .env and update with with your TELNYX_API_KEY and TELNYX_PUBLIC_KEY

Run the webhook server:

bash

python main.py

In a separate terminal, expose it via ngrok so Telnyx can reach it during development:

bash

ngrok http 8000

Copy the ngrok HTTPS URL, then open the assistant in the Telnyx portal and paste it Dynamic Variables webhook URL field:

[

](https://x.com/Sumanth_077/article/2076671269391696042/media/2076668429046816768)

Three ways to test the agent:

Portal test. Open the assistant in the Telnyx portal and click Test. A browser-based call interface opens. No phone required.
Phone call. Dial the number you attached to the assistant. Real PSTN, real audio, real telephony codecs.
Browser widget. Open demo.html from the repo in any browser. It embeds the Telnyx AI Agent widget, which connects to the assistant over WebRTC. No server required.

Sample prompts that shows what this agent can handle:

"I think my debit card was stolen. What should I do?" Grounded answer from the KB, with the 24-hour hotline number and the app-based card freeze option.
"How long does a wire transfer take, and what does it cost?" Same source, same call.
"How busy are you right now?" Answered from the live queue wait injected via the webhook.

The full setup and code is available

here

.

What you get from this

Voice AI used to mean stitching together a speech-to-text service, an LLM, text-to-speech, and then writing the code to keep everything in sync. Every new integration added more moving parts to manage.

With Telnyx, all of that is handled for you. The only code left is the webhook that answers one question: What does the agent need to know right now? That lets you focus on your business logic instead of the infrastructure.

Checkout the Repo:

https://github.com/Sumanth077/Hands-On-AI-Engineering/tree/main/audio/customer_support_voice_agent