Multi-Domain Support Triage Agent

Industry

SaaS · Support Automation

Business outcome

96.6% evaluation agreement on a 30-question gold-set review, using manual validation and LLM-as-judge scoring against expected answers and referenced knowledge-base sources.

Stack

Python · Google ADK · Gemini 2.5 Flash · ChromaDB · Pydantic · Streamlit · 774 articles · 6,454 vectors · fully local retrieval

Support operations were slowed by manual ticket routing and inconsistent triage quality, creating avoidable response delays.

Designed and implemented a reliable triage pipeline with confidence gating and escalation logic. A 30-question gold-set review reached 96.6% evaluation agreement through manual validation and LLM-as-judge scoring.

System Design

Built one shared core engine used by both CLI batch processing and a Streamlit operations interface, reducing maintenance overhead and keeping behavior consistent across workflows.

Introduced a deterministic retrieval-confidence gate before model invocation so routing decisions are policy-driven and predictable, not prompt-dependent.

Implemented dual-path handling: low-confidence tickets are escalated immediately, while high-confidence tickets continue through a Google ADK sequential flow — Retrieval Agent identifies the correct knowledge domain and pulls evidence from an isolated ChromaDB index; Format Agent enforces a strict Pydantic schema for response, status, and citation justification.

Reliability Strategy

• Reduced operational risk by separating routing control from text generation, preventing the model from answering when evidence quality is weak.
• Enforced structured outputs with Pydantic to improve downstream reliability for support tooling and analytics.
• Added graceful-failure escalation reasons to improve human handoff quality and speed up agent resolution.
• Chunked and embedded 774 knowledge articles (HackerRank, Claude, Visa) into 6,454 ChromaDB vectors for fully local retrieval with no cloud dependency.
• Evaluated SBERT vs EmbeddingGemma on a 10-ticket sample — chose EmbeddingGemma for stronger citation alignment on ambiguous and multi-hop evidence despite slower inference.

Tradeoffs

• Prioritized trust and consistency over broader recall by limiting retrieval to verified internal policy content in ChromaDB.
• Chose deterministic distance-threshold gating over model self-evaluation to make safety behavior explainable and auditable.
• Accepted threshold-tuning effort as an explicit investment to gain predictable escalation behavior during pilot usage.
• EmbeddingGemma accuracy over SBERT speed — wrong citations mean wrong customer responses.
• Sequential agents over single-step pipeline — reliability and debuggability over latency.