Real-Time AI Phone Calls That Scale on AWS Fargate

Challenge

Evrella, the AI client services assistant we build and run, answers businesses' phone calls. A call is not a web request: it is a live, two-way audio stream that lasts minutes, and every turn chains speech-to-text, an AI reply and text-to-speech while the caller waits. The service began on a single rented server — one point of failure and a fixed ceiling on calls — and had to be sized for 100 calls at once from day one, without paying for idle capacity or cutting anyone off mid-sentence.

Approach

• Moved the voice service onto Amazon ECS on Fargate (ARM64) behind an Application Load Balancer, with a one-hour idle timeout so the network never drops a long call
• Made every container stateless: the incoming-call webhook, the audio WebSocket and the after-call callback can each land on a different task, so a call's state lives in DynamoDB and a conditional write claims it exactly once
• Scaled on live calls, not CPU: each task reports its active calls every 30 seconds, and a task that is stopping takes no new calls and lets the live ones finish first
• Kept serverless where it fits: the AI brain stays on Lambda with reserved concurrency, so a burst of web chats cannot starve the phone line, and after-call work — the SMS, the call report, the quality review — goes through SQS with a dead-letter queue
• Isolated through IAM rather than network plumbing: a minimal VPC with no NAT gateway, a task role that signs every request to the AI brain, and secrets read from Secrets Manager — never baked into the image
• Cut over with a way back: the old server stayed ready behind a 60-second DNS TTL during the switch

Results

Technologies