A phone receptionist that drops a caller loses a customer, and a voice agent’s tools fail in ways a chat agent’s don’t: the caller hangs up mid-sentence, a setting is missing, or the phone network refuses a transfer while someone is listening. Building our own AI receptionist taught us that every tool needs a safe fallback that still captures the caller, and that telephony features need proof on live calls, not just passing unit tests.
The receptionist answers our main line. It answers basic questions about us, records new callers as leads in our CRM, opens support cases for existing clients, and checks office hours and holidays. It uses Azure Communication Services for the phone number and Call Automation, the Azure Voice Live API with a realtime voice model for speech, and Azure Container Apps for the backend. It is still evolving, and this post covers what isn’t finished.
Every tool must degrade safely
On a call, a failed tool sounds like a pause, or nothing. So each tool follows one rule: whatever goes wrong, the caller’s request still reaches a person.
| Failure | What happens |
|---|---|
| Caller hangs up mid-intake | On shutdown, the call handler posts a lead with the caller ID and the call transcript |
| Transfer requested, no number configured | The tool returns “not configured” and the agent takes an urgent message |
| Transfer fails or the number is bad | The agent apologizes and takes a message, without retrying |
| No support-case token configured | The request is recorded as a lead, tagged with its urgency |
| Client has no support agreement | A lead is recorded, not a case, with the scope of what they asked for |
The hang-up fallback took the most care. If the caller spoke but no lead was submitted, the handler posts one named “Unknown caller”, with the transcript in the notes; our CRM rejects leads with no name, email or company. Calls with no caller speech, or no caller ID, are skipped. The notes are trimmed to stay under the CRM’s 4,000-character limit.
Tool results also tell the model what to do next, in plain words. A failed transfer returns success: false plus an instruction to take a message and tell the caller the team will be alerted, and the prompt says not to try again. Given only a raw error, a model may retry or invent a reason.
Recognizing callers without widening access
Existing clients shouldn’t have to spell their email to a robot. When the caller first speaks, a lookup tool matches the caller ID against client phone numbers in the CRM, and the match is cached for the rest of the call. A recognized client is greeted by name and company, and the request goes to the right place for their support tier.
- Compare the last 10 digits. Numbers arrive with or without a country code, brackets, dashes or spaces. We strip everything but digits and compare the last ten; anything shorter doesn’t match at all.
- Use a separate, read-scoped key. Creating a lead uses a public web-form token, like the one our website’s contact form embeds, which can only create leads. Looking up clients uses a different, read-scoped key stored as a secret in Key Vault. If it is missing, nobody is recognized and every caller is treated as new, which is the safe direction to fail.
Normalize messy configuration
Support tiers come from a custom field on each client record, typed by people. “Tier 1”, “tier-1”, “TIER1” and “1” should all mean the same thing, so the code normalizes every tier name before use:
def tier_key(tier):
key = "".join(ch for ch in str(tier or "").lower() if ch.isalnum())
return f"tier{key}" if key.isdigit() else key
All four examples become tier1. A tier that still isn’t recognized falls back to the default, lowest tier and logs a warning, rather than failing the case; staff can re-tier it later. Malformed JSON in a settings map is logged and ignored rather than treated as fatal.
Tune the conversation for how people actually talk
- Silence threshold. We raised the end-of-turn silence threshold to 800 ms. Callers pause while spelling an email address, and the agent shouldn’t take that pause as the end of their turn.
- Read-back. The agent spells email addresses back letter by letter and corrects them until the caller confirms. It never guesses a spelling.
- Guardrails. It never quotes prices, rates or estimates. If asked whether it is a person, it says plainly that it is an AI assistant that takes messages for the team. It never asks for payment details or passwords, and the greeting says the call may be recorded and transcribed.
Verify what unit tests can’t prove
Live transfer is the clearest example. The transfer tool calls the Call Automation transfer API, and its unit tests stub that call. They prove our routing and fallback logic. They can’t prove that a real inbound call reaches the destination, that audio flows both ways, or that the receptionist stops talking when the call bridges.
So transfer shipped switched off; with no destination number, the agent takes a message. Before we relied on it, our runbook required:
- two consecutive clean live transfers on a deployed environment, with the expected caller ID at the destination and no errors in the logs
- both fallback paths checked on a live call: no number configured, and an unreachable number
- a caller hanging up mid-transfer, with the end-of-call fallback still firing
We have now run that verification on live calls. Transfer stays off by default and is turned on per deployment by setting a destination number. Rollback is the same configuration change in reverse: unset the number.
Known gaps
- No warm transfer. Transfers are cold. The tool captures a summary for the person receiving the call, but nothing speaks it to them yet.
- It can’t hang up. The caller always ends the call; the agent can’t end it yet.
- Floating holidays are manual. Fixed-date holidays are built in; ones that move each year are added by hand.
How we can help
If you are considering an AI receptionist, the hard parts are the ones above: tool failures, and proving telephony behavior on real calls. We run ours on our own line first, and our AI receptionist page describes how we offer it.
