Voice Concierge

The phone line is the bottleneck, not the doctors.

Most of what front-desk staff and call-center agents field every day is the same handful of questions, asked at the wrong hour for a human to answer.

“What are your visiting hours?” “Is Dr. Perera in on Tuesday?” “I need to move my 3pm.” “What should I bring for a CT scan?” None of these need a clinician — but today they all compete for the same front-desk headcount as genuine emergencies, and none of them get answered after the desk closes.

A hospital’s own knowledge — department hours, doctor schedules, prep instructions, policies — already exists in documents and systems. The gap isn’t information, it’s access: a patient calling at 9pm, or typing instead of waiting on hold, has no way to reach it without a person on the other end.

This project is a pattern for closing that gap: a voice and chat assistant that answers from the hospital’s real data — not a generic chatbot guessing from the internet — and that knows the difference between a question it can safely answer and one that needs a human.

One assistant, two front doors, one brain.

A patient can call it or type to it. Either way, the same reasoning system answers — grounded in live scheduling data and the hospital’s own documents, not improvisation.


VOICE Phone-call-grade conversation	CHAT Text assistant with the same brain	SCHEDULING Real bookings, not a form
Streams speech to text, thinks, and speaks back — first words heard in about 1.5 seconds, with natural interruption handling.	The web chat widget shares the identical routing and memory system — a richer, citation-style answer for readers instead of listeners.	Looks up a patient’s record, finds a doctor, and creates, reschedules, or cancels an appointment against the live system — not a script.
KNOWLEDGE Grounded medical-information answers	MEMORY Remembers the patient, not just the sentence	GUARDRAIL Knows its lane
Answers about departments, procedures, and policy are retrieved from the hospital’s own knowledge base, then written into a plain-language reply.	Short-term recall inside one call, long-term facts across visits (“prefers evening appointments”), and an honest record of what was actually said if interrupted.	Politely declines questions outside hospital business — trivia, other companies, unrelated requests — and redirects to the main line instead of guessing.
ESCALATION Live web lookup as a fallback	OVERSIGHT Every turn is auditable	COST Scales down when idle
For time-sensitive general questions the knowledge base doesn’t cover, a real-time search tool fills the gap rather than inventing an answer.	Each reply is traced end-to-end — which route handled it, what data it used, how long it took, what it cost — visible to the team running the system.	Common questions are answered from a cache in under 300ms; the voice service itself spins down to zero between calls instead of running around the clock.

What happens between “hello” and an answer.

Six steps, mostly invisible to the caller, tuned so the pauses feel like a person thinking rather than a system loading.

The system notices you’ve stopped talking — Voice-activity detection watches the audio stream and marks the end of a turn about 0.3 seconds after silence — fast enough to feel responsive, slow enough not to cut people off mid-thought.
Speech becomes text as you speak — A streaming speech-to-text service transcribes continuously, so there’s no separate “processing” pause once you finish a sentence.
Two checks run at once — A scope guardrail asks “is this hospital business?” while a router asks “who should answer this — the front desk, the clinical knowledge base, or a direct reply?” Running them in parallel hides one behind the other.
The right specialist answers — with real data — The chosen agent pulls from the live scheduling system or the hospital’s own documents before writing a reply, so it can’t invent a doctor, a time, or a policy that doesn’t exist.
The reply streams back as speech — Text-to-speech starts speaking the first words while the rest of the sentence is still being generated — the caller isn’t waiting for the full answer before hearing anything.
If you cut in, it remembers what actually happened — An interruption stops playback immediately, and only the words that were truly spoken are kept in memory — so a transcript or callback never claims something the caller never heard.

Architecture — simplified

flowchart LR
    P(["Patient<br/>voice call or chat"]) --> GR{{"Guardrail + router<br/>in scope? · which agent?<br/>~150–800ms, in parallel"}}
    GR -.->|"common FAQs<br/>~0.3s"| CACHE[("Instant-answer cache")]

    subgraph AGENTS["Specialist agents"]
        direction TB
        FD["Front-desk agent<br/>bookings · records"]
        CL["Clinical agent<br/>hospital knowledge base"]
        CO["Concierge agent<br/>small talk · live web"]
    end

    GR --> FD
    GR --> CL
    GR --> CO
    CACHE -.-> M(["Merge & respond<br/>streams back to caller<br/>≈1.5s total, voice path"])
    FD --> M
    CL --> M
    CO --> M
    GR <--> MEM[("Memory<br/>this call · past visits ·<br/>recalled facts · learned habits")]
    M --> MEM
    M --> P

    classDef store fill:#eef3fb,stroke:#2E5496,color:#1F3864;
    class MEM store;
    classDef cache fill:#fff4e6,stroke:#ff7e33,color:#7a3b00;
    class CACHE cache;

Simplified from the working system: a supervisor graph fans a request out to one or more specialist agents in parallel, then merges their answers into one reply. Common FAQs skip the agents entirely via a cache. Every stage reports to a trace log (Langfuse).

Latency budget (measured)

≈1.5s — silence to first spoken word, on the voice fast path

Stage	Endpointing	Speech→text	Route + fetch	First LLM token	Text→speech	Network
Time	300ms	200ms	<150ms	200–400ms	200ms	200ms

Who touches it, and how

Four roles, one system.

Role	What they touch	What changes for them
Patient	A “Call” button in the patient app, or the chat panel beside it	Can ask about hours, doctors, and their own bookings at 2am and get an immediate, accurate answer — or be told plainly when it can’t help and given the main line
Front-desk & call-center staff	Nothing new to operate — the assistant handles the repetitive volume upstream	Fewer routine calls reach a human queue; the ones that do are the ones that actually need a person
IT & operations	A trace dashboard showing every conversation: route taken, data used, latency, cost	Can audit any single answer after the fact, and see usage and spend trending in real time
Leadership	A cost dial — the voice service can be paused, scaled, or torn down independent of the text assistant	Coverage can start with text chat only, add voice for a pilot department, and scale spend with adoption instead of committing to it up front

What a demo session actually shows, end to end:

Channel	What’s asked	What it proves
Chat	“What are the opening hours?”	Cached instant answer, ~0.3s
Chat	“Do I have a booking next week?”	Live lookup against the patient’s own record
Voice	“Book me an appointment with a cardiologist”	Full voice turn, real scheduling data, ≈1.5s to first word
Voice	“Tell me about post-surgery care…” (interrupted mid-sentence)	Playback stops instantly; memory keeps only what was actually said

Trust & safety

What stops it from making things up.

The riskiest failure mode for a hospital assistant isn’t refusing to answer — it’s answering confidently and wrongly. The system is built to prefer the first over the second.

SCOPE — Stays in its lane. A dedicated check runs on every message and declines anything outside hospital business — general trivia, unrelated products, requests to role-play — with a polite redirect to the main line, before any real answer is attempted.
GROUNDING — Answers from records, not recollection. Scheduling and clinical-information questions are answered from the hospital’s live systems and knowledge base. When the source data doesn’t contain the answer, the assistant says so instead of guessing at a name, time, or policy.
FAILS OPEN — A provider outage never locks a patient out. If a safety check itself fails, the system defaults to letting the normal, cautious answer path continue rather than refusing service — a technical hiccup shouldn’t become a patient’s problem.
HONESTY — Remembers what happened, not what was intended. An interrupted reply is stored tagged as partial, and long-term memory is never built from a cut-off answer — so a later transcript or callback can’t misrepresent the conversation.
DATA MODEL — Built against a real hospital privacy framework. The reference knowledge base the assistant is trained to reason over models role-based record access, consent, breach response, and retention in line with PDPA/HIPAA-style principles — so the assistant’s judgment about what’s confidential is grounded in policy, not improvised. This is a modelled framework in the prototype, not a compliance certification of the software itself.
AUDITABILITY — Nothing happens off the record. Every turn — text or voice — produces a full trace: which agent answered, what data it touched, how long each step took, and what it cost. Any answer can be reconstructed and reviewed after the fact.

Under the hood

Off-the-shelf pieces, purpose-built wiring.

Nothing here is a bespoke model — the value is in how proven, specialised services are routed together and grounded in the hospital’s own data.

Layer	Technology	Role
Call transport	`LiveKit Cloud`	Real-time audio between the caller’s browser and the assistant, over WebRTC
Speech in	`Deepgram Nova-3`	Streaming transcription — text appears as the caller speaks
Speech out	`ElevenLabs Turbo v2.5`	Streaming, natural-sounding voice synthesis
Reasoning (voice)	`Groq · Llama 3.3 70B`	Sub-second streaming replies tuned for the live call
Reasoning & orchestration (chat)	`LangGraph multi-agent graph`	Routes a question to the right specialist, or several in parallel, then merges the results
Tool access	`Model Context Protocol (MCP)`	A standard interface connecting agents to hospital systems and data — swappable without rewriting the agents
Records & memory	`Supabase (Postgres + pgvector)`	Scheduling data, chat history, and long-term patient facts
Knowledge search	`Qdrant`	Vector search over the hospital’s knowledge base and cached answers
Oversight	`Langfuse`	Per-turn tracing, latency, and cost tracking across every conversation
Infrastructure	`AWS Fargate · GitHub Actions`	Serverless containers behind one load balancer; push-to-deploy CI/CD

Status & running cost

What it costs to keep the lights on.

$0/day	0 → 1	$20
idle cost floor when the whole stack is torn down between pilots	voice containers scale on demand per call, not kept running 24×7	spend increment that triggers an automatic budget alert

The system runs on three small serverless services — the API and web app, a background worker for slow jobs like memory summarisation, and the voice service — behind a single load balancer, on ARM-based cloud instances chosen for their cost efficiency. Only the voice service costs anything while nobody is calling: it scales to zero and starts on demand for a call or a demo. A paused environment still carries a small fixed cost for the load balancer and managed database; a full teardown reaches zero and rebuilds in about 15–20 minutes.

Honest disclaimer

This is a working engineering prototype built end-to-end — including a synthetic knowledge base, scheduling records, and patient data — as a demonstration of the pattern, not a live deployment inside a hospital’s real systems. The latency and cost figures above are measured against this sandbox. Before any real rollout: connect it to the hospital’s actual scheduling/EMR systems, have clinical staff review and sign off on the knowledge base content, and run a formal security and compliance audit.