How to Build an AI Support Agent That Knows When NOT to Answer Tickets

                     How to Build an AI Support Agent That Knows When NOT to Answer Tickets
               



Tópico:
                     How to Build an AI Support Agent That Knows When NOT to Answer Tickets
               
Categoria: Tutoriais | FreeCodeCamp Premium
Idioma Principal: Português (Conteúdo de Tecnologia)

Conteúdo do Tutorial / Guia Passo a Passo:
-------------------------------------------------------------------------
Most AI support agent tutorials show you how to wire up Retrieval Augmented Generation (RAG) and call it a day. Convert the docs into numeric vectors, pull the closest few passages to the user's question, drop them into a prompt, and ship a polite reply.

This pattern works for FAQ tickets, but it breaks the moment a user writes "my card was stolen", for example. The agent confidently quotes an outdated phone number, the user loses minutes which matter, and the support team finds out from a complaint.

I'm a full-stack software engineer working with fintech systems. I shipped a multi-domain triage agent for the HackerRank Orchestrate hackathon, a 24-hour solo build judged across four axes. The agent handled real support tickets across HackerRank, Claude, and Visa, grounded only in the documentation provided with the starter repo. Two of those domains tolerate a wrong answer. The third does not. I ranked 9th of 1,349 participants on the final leaderboard. The full source is on GitHub.

This article walks through the pattern I used to keep the agent safe: escalation-first design. The agent commits its routing decision before any text is generated, drafts grounded answers only when the routing says reply, and verifies the answer with two independent AI judges before it reaches the user. Every step is built to fail toward escalation, not toward a wrong answer. I also walk through the gaps in my own submission, so you don't repeat them.

What you'll find below:

• Why letting the language model make the escalation decision is the wrong default

• The pure-function decider pattern and its three terminal paths

• A two-judge consensus verifier with an arbiter for disagreement

• How to make all of this cheap with Jaccard pre-checks and SHA-keyed caching

• Five honest gaps in my own submission, and what I would change next time

Table of Contents

• The Two Halves of Support Tickets

• Why Letting the LLM Decide Is the Wrong Default

• The Pure-Function Decider Pattern

• Three Terminal Paths Instead of Two

• The Consensus Verifier as a Second Safety Net

• Cost and Observability

• Where I Got It Wrong

• Five Gaps I Would Close in a Rematch

• Where This Pattern Belongs

The Two Halves of Support Tickets

Support tickets aren't one problem. They are two.

Most tickets are FAQs. "How do I add time accommodation for a candidate?" or "How do I delete a conversation in Claude?" These have direct answers in the documentation. An AI agent resolves them in seconds and frees the human team for harder work. This is the more obvious half.

A small fraction of tickets are sensitive. "My Visa card was stolen." "I want to appeal my test score." "Please delete all my data." On these, an AI confidently giving a wrong answer is worse than no answer at all. It delays the real human response. It causes real harm to the user. This is the harder half.

The design problem is not "build a chatbot." It's "build something that knows the difference between the two and route accordingly". The whole architecture below exists to enforce this routing reliably:

In the diagram above, you can see that tickets fan out to triage signals and retrieval, then feed a Python decider with no LLM call. The decider routes to one of three paths: escalate to a human, send a template decline for off-topic requests, or hand off to the drafter for a grounded answer with citations. Drafts pass a cheap token-overlap check first. Safe high

... [O tutorial continua no link abaixo] ...