SaaS · Support Automation
Multi-Domain Support Triage Agent
Owned end-to-end delivery of an AI ticket triage system with domain-scoped RAG and deterministic confidence gating — 96.6% citation grounding on a 29-ticket HackerRank Orchestrate eval.
- Outcome
- 96.6% citation grounding across 29 consecutive Orchestrate eval tickets — zero hallucinations, with deterministic confidence gating that blocks low-evidence AI responses before they reach customers.
Stack & scale
Python · Google ADK · Gemini 2.5 Flash · ChromaDB · Pydantic · Streamlit · 774 articles · 6,454 vectors · fully local retrieval
Support operations were slowed by manual ticket routing and inconsistent triage quality, creating avoidable response delays.
Designed and implemented a reliable triage pipeline with confidence gating and escalation logic — 96.6% grounding on a 29-ticket Orchestrate eval with zero hallucinated responses.
System Design
Built one shared core engine used by both CLI batch processing and a Streamlit operations interface, reducing maintenance overhead and keeping behavior consistent across workflows.
Introduced a deterministic retrieval-confidence gate before model invocation so routing decisions are policy-driven and predictable, not prompt-dependent.
Implemented dual-path handling: low-confidence tickets are escalated immediately, while high-confidence tickets continue through a Google ADK sequential flow — Retrieval Agent identifies the correct knowledge domain and pulls evidence from an isolated ChromaDB index; Format Agent enforces a strict Pydantic schema for response, status, and citation justification.
Reliability Strategy
- Reduced operational risk by separating routing control from text generation, preventing the model from answering when evidence quality is weak.
- Enforced structured outputs with Pydantic to improve downstream reliability for support tooling and analytics.
- Added graceful-failure escalation reasons to improve human handoff quality and speed up agent resolution.
- Chunked and embedded 774 knowledge articles (HackerRank, Claude, Visa) into 6,454 ChromaDB vectors for fully local retrieval with no cloud dependency.
- Evaluated SBERT vs EmbeddingGemma on a 10-ticket sample — chose EmbeddingGemma for stronger citation alignment on ambiguous and multi-hop evidence despite slower inference.
Tradeoffs
- Prioritized trust and consistency over broader recall by limiting retrieval to verified internal policy content in ChromaDB.
- Chose deterministic distance-threshold gating over model self-evaluation to make safety behavior explainable and auditable.
- Accepted threshold-tuning effort as an explicit investment to gain predictable escalation behavior during pilot usage.
- EmbeddingGemma accuracy over SBERT speed — wrong citations mean wrong customer responses.
- Sequential agents over single-step pipeline — reliability and debuggability over latency.
Demo
[ SCREENSHOTS ]