Overview

# The Orchestration Layer Is Where Voice AI Agents Actually Work Your voice AI agent works perfectly in testing. In production, it's a liability—not because the STT is wrong or the LLM hallucinates, but because you're treating orchestration as an afterthought instead of the system that actually runs the show. The voice AI market is unbundling. Deepgram's messaging, Twilio's pivot toward modular stacks, and a dozen smaller players are all converging on the same narrative: stop buying monolithic platforms. Buy components. Compose them yourself. You'll get better pricing, more flexibility, and control over your stack. That narrative is correct. And it's incomplete. What the unbundling story misses is that enterprises choosing modular voice stacks aren't actually solving the hard problem—they're just making it visible. The hard problem is orchestration: the logic layer that sits between your LLM and the actual production system, deciding which provider to use, when to retry, how to handle ambiguity, when to escalate, and how to keep the whole thing profitable at scale. When you buy a monolithic platform like Twilio Studio or Five9, you're not paying for the STT or TTS. You're paying for orchestration that someone else built, tested, and is now running in production for thousands of customers. You're paying for the fallback logic when Deepgram times out. You're paying for the cost optimization that routes simple calls through a cheap provider and complex ones through a better one. You're paying for the monitoring that tells you when something is wrong before your customers do. The moment you move to a composable stack, you inherit that entire problem. ## The Orchestration Layer Is Where Production Fails Consider a straightforward scenario: an AI agent handling inbound customer support calls. Your architecture looks clean on the whiteboard: - Twilio for PSTN ingestion and audio streaming - Deepgram for speech-to-text - OpenAI for reasoning - ElevenLabs for text-to-speech - Your own backend for business logic (CRM lookup, ticket creation, etc.) This is a real setup. It's modular. It's flexible. It's also incomplete. Now add production constraints: **Latency.** Deepgram's STT takes 200ms on average. Sometimes it takes 1.2 seconds. Your LLM inference adds another 800ms. ElevenLabs TTS adds 300ms. Total: 2.3 seconds from speech end to audio start—sometimes 4 seconds. Your customer is waiting. If you cross 3 seconds, they think the line is dead. Who manages that timeout? Who decides whether to use a faster, lower-quality STT when latency spikes? That's orchestration. **Failure modes.** Deepgram is down. What do you do? Switch to Google Cloud Speech-to-Text? That requires a contract, credentials, monitoring. Who decides the switching logic? Who tests it? Who verifies it actually works when Deepgram is actually down (not in your staging environment, but at 2 AM on a Tuesday when you have 50 concurrent calls)? That's orchestration. **Cost.** Deepgram's per-minute pricing is $0.0043. Google Cloud Speech-to-Text is $0.024 per 15 seconds. OpenAI's GPT-4o is $15 per million input tokens. ElevenLabs is $0.30 per 1K characters. A 10-minute call with 5,000 characters of TTS and 2,000 tokens of LLM reasoning costs roughly $0.18 in Deepgram + $0.30 in ElevenLabs + $0.03 in OpenAI. But if you're running 10,000 calls a day, that's $5,400 a day, or $1.97M annually. At that scale, you need routing logic that says: "For calls that are clearly spam, use the cheapest STT and fastest LLM. For complex support cases, use the best models." Who implements that? Who monitors whether it's actually saving money or just degrading quality? That's orchestration. **Quality.** Your STT is confident 95% of the time. 5% of the time, it's guessing. Your LLM catches some of those guesses ("Did you say X?"), but not all. How do you handle ambiguous speech? Do you ask for clarification? Route to a human? Retry with a different STT provider? Do you have SLAs on first-call resolution? Who tracks whether the orchestration logic is meeting them? That's orchestration. **Observability.** You're running 50 concurrent calls across 5 providers. One of them is degraded. Your customers aren't complaining yet, but they will be in 20 minutes. How do you know it's happening? Do you have latency thresholds per provider? Cost anomaly detection? Quality regression alerts? Who's watching? That's orchestration. None of these problems are solved by choosing Deepgram over Google Cloud or ElevenLabs over Google Cloud TTS. They're solved by building an orchestration layer that treats the modular components as resources to be routed, monitored, and optimized in real time. ## Why Orchestration Is the New Moat This is why the unbundling narrative is only half the story. When voice AI was bundled (Twilio Studio, Five9, Genesys), the competitive moat was the platform itself—the breadth of integrations, the UX, the ecosystem lock-in. Now that it's unbundled, the moat has shifted. Deepgram's STT is good, but it's not defensible. ElevenLabs' TTS is good, but OpenAI's is getting better. The LLM is a commodity. The moat is now the orchestration layer—the system that knows how to route calls across providers, optimize for cost and quality simultaneously, handle edge cases, and scale to thousands of concurrent calls without degrading. This is exactly what we're seeing in the broader market. n8n is shipping MCP integrations not because they're building a better MCP client, but because they're building the orchestration layer that makes it possible to compose APIs reliably. Atlassian is shipping agent tooling not because they're building a better LLM, but because they're building the orchestration layer that makes it possible to embed agents into Jira workflows. Google is shipping containerized execution environments not because they're building a better runtime, but because they're building the orchestration layer that makes it possible to run agent code at scale. In voice AI, the same shift is happening. The next wave of competitive advantage isn't in the STT model or the TTS quality. It's in the orchestration layer that routes, monitors, and optimizes across providers at production scale. ## What This Means for Your Voice AI Agent If you're building a voice AI agent today, you have three options: **Option 1: Buy a bundled platform.** Use Twilio Studio or Five9 or Genesys. Accept that you're paying for orchestration you don't need and that you're locked into their component choices. This is the right call if you have simple requirements and you want someone else to worry about production. **Option 2: Build a modular stack and ignore orchestration.** Use Deepgram + OpenAI + ElevenLabs + whatever you want. This is what most teams do. It works until it doesn't—usually around the time you hit 100 concurrent calls or your first major outage. Then you realize you've built a house of cards. **Option 3: Build a modular stack and invest in orchestration.** This is the right call if you're serious about voice AI and you want to own your stack. But it means treating orchestration as a first-class problem, not an afterthought. It means building (or buying) a layer that handles routing, fallback, cost optimization, quality monitoring, and observability. It means accepting that orchestration is 60% of the work and the component selection is 40%. Most teams choose Option 2 and regret it. Some teams realize the cost and complexity of building orchestration and move to Option 1. The teams that succeed are the ones that choose Option 3 early and treat orchestration as the core product. ## The Orchestration Layer Is Shipping Now The good news: the infrastructure for production voice orchestration is starting to exist. Deepgram is adding fallback logic. Twilio is exposing more granular routing controls. Open-source projects like Vapi are building orchestration layers specifically for voice agents. Cloud providers are adding managed voice orchestration services. But most of these are still nascent. Most teams building voice AI today are either accepting the bundled platform tax or building orchestration from scratch. If you're in that second camp, here's what you need to think about: 1. **Latency budgeting.** Define your acceptable latency per component. Build monitoring that tracks it. Set up alerts when you're approaching your budget. Build fallback logic that triggers when you exceed it. 1. **Provider fallback.** Don't assume your primary provider is always available. Build a secondary provider into your stack. Test it regularly (not just in staging). Monitor whether it's actually working when you need it. 1. **Cost routing.** Map your call types to optimal providers. Simple calls go through cheap providers. Complex calls go through expensive, high-quality ones. Track whether you're actually saving money. 1. **Quality tracking.** Define what "quality" means for your use case (first-call resolution, customer satisfaction, whatever). Track it per provider. Alert when it degrades. 1. **Observability.** Log everything. Latency per component. Cost per call. Quality metrics. Provider performance. Build dashboards that show you what's happening in real time. This is work. It's not glamorous. It's not the part of voice AI that gets written about in TechCrunch. But it's the difference between a voice agent that works in testing and one that works in production. The unbundling of voice AI is real and it's right. But the narrative that modular components solve the hard problem is incomplete. The hard problem is orchestration. The teams that solve it will own the market. The teams that ignore it will spend the next two years debugging production failures and wondering why their voice agents don't work.