Before the model
Policy pre-routing
Emergencies, opening hours, and handoff to a human are decided by policy, not by the LLM. These cases cost zero tokens and behave the same every time.
← Back to AI agent projects
Case study · Reliability · Guardrails · Cost & latency
Agent Receptionist
A tool-calling LLM agent for appointment-based businesses, built as an engineering exercise in making an agent predictable: cited answers, guarded output, a deterministic fallback, and latency and cost I can measure.
deterministic eval
44/44
live Groq eval
43/44
live question test
19/20
per question (one live check)
~$0.0002
Watch the walkthrough. A 73 second walkthrough with voiceover: the case study, the live demo (tool calling, RAG with sources, latency and cost per turn) and the same engine on a different fictional business.
01 · problem
Problem
Appointment businesses need answers that are fast and correct, not just fluent.
A front-desk agent for a hotel or a dental practice is a small, high-stakes domain. A wrong opening hour, an invented service, or a casual reply to an emergency is worse than no answer. A plain "LLM plus prompt" can do all three.
I built Agent Receptionist to see how far an LLM agent can be pushed toward reliable behaviour: ground answers in the business's own documents and cite them, handle the predictable cases without the model at all, check every sentence before it reaches the user, keep a deterministic path for booking, and measure latency and cost instead of guessing.
The two demo businesses, a hotel and a dental practice, are fictional. They exist to exercise the same engine against different documents, hours, and policies. Switch between them with ?biz=hotel and ?biz=dental.
02 · architecture
Architecture
Cheap, deterministic decisions come first. The model is called only when it is needed, and its output is checked before it streams.
React + Vite clientDashboard per business (
?biz=). Renders streamed text, citations, and the latency/cost panel.No secrets in the client→
Node / Express APIReceives the question, orchestrates the steps below, and streams the reply back over SSE.Raw provider errors stay here
→
1. Policy pre-routerEmergencies, opening hours, and human handoff are routed by rules.Zero tokens
2. Tool-calling agentGroq,
openai/gpt-oss-120b. Calls tools and answers from retrieved context.RAG with cited sources→
3. Per-sentence guardEntity allowlist, hours check, required-fact check, applied sentence by sentence.
→
4. SSE stream + panelTokens stream to the client as they are produced. The panel shows latency and token cost per question.
Fallback: deterministic booking engineBooking logic that does not depend on the model, kept as the fallback path alongside the agent.
Stack: React, Vite, TypeScript on the client; Node and Express with SSE on the server; deployed on Vercel.
03 · reliability
Reliability & guardrails
Layers around the model, each cheaper and more predictable than the model itself.
Grounding
RAG that cites sources
Answers come from the business's own documents and carry their sources, so a reader can check where a claim came from instead of trusting the model.
After the model
Per-sentence guard
- Entity allowlist: names and services must be known to the business.
- Hours: stated times are checked against the real schedule.
- Required facts: answers that must contain a fact are checked for it.
Fallback
Deterministic booking engine
Booking does not have to depend on model behaviour. A deterministic engine is kept as the fallback path, so the agent is not the only thing standing between a user and a booking.
Security
Secrets and errors stay server-side
- No provider secrets in the client.
- Raw provider errors are kept server-side and are not shown to users.
Delivery
Streaming and visibility
Replies stream over SSE with real token streaming, and a latency and cost panel makes each question's timing and token usage visible rather than assumed.
04 · results
Measured results
From my own test runs. Live runs were on 2026-10-24.
| Check | Result | Notes |
|---|---|---|
| Deterministic eval | 44/44 | Hotel 23/23, dental 21/21. |
| Live Groq eval | 43/44 | 2026-10-24. One eval-regex miss: the check's regex was too narrow; the reply itself was right. |
| Live 20-question test | 19/20 | 2026-10-24. One eval-regex miss, same cause: too-narrow regex, correct reply. |
| Streaming events | 11 median | Real token streaming, not a single buffered response. |
| Time to first text | ~550 ms | Of ~610 ms total, in tests. |
| Live public-URL check | ~840 ms first token | 563 prompt + 251 completion tokens, about $0.0002 per question. A single check, not a benchmark. |
| Policy-routed replies | 0 tokens | Emergencies, hours, and handoff never reach the model. |
Small suites on fictional businesses: these numbers show the engine behaves as designed, not how it would perform on a real business's traffic. The latency and cost figures come from a handful of runs and one live check.
05 · tradeoffs
Tradeoffs & what I'd improve
What this design gives up, and what I would do next.
Tradeoffs
Predictability over flexibility
- Policy routing and the guard make behaviour checkable, but they limit how freely the agent can phrase or improvise.
- Allowlists and fact checks need maintaining per business as documents change.
- A deterministic fallback adds a second code path to keep correct.
Next
What I'd improve
- Fix the narrow eval regex and make live checks less brittle than pattern matching.
- Larger live suites and repeated latency runs, instead of one live check.
- Test on real business documents, not only the two fictional ones.
06 · live demo
Live demo
Try it on either fictional business.
Open the dashboard and ask a question, then look at the citations and the latency/cost panel. Try asking about an emergency or opening hours to see policy pre-routing.
Source code is private. Available on request or in interviews: usmamalik346@gmail.com.