A support assistant that cites its sources and knows when to stay quiet

The client's support team handled about 12,000 tickets a month: routine product questions mixed in with billing disputes, sensitive account changes and the occasional incident. Answers to the routine ones existed, spread across product docs, internal runbooks and tenant-specific records. Experienced agents knew which source to trust. Newer ones could find a plausible document without noticing it described an older product version or a different customer's entitlement.

A chatbot would have made search look easier and the access problem much worse. One fluent answer could contain unsupported advice, leak another tenant's data or suggest an action the agent wasn't allowed to take. So we built assistance inside the existing workflow. The assistant drafts with citations, and a person decides what gets sent.

Permissions come before retrieval

Every request carried the agent's authenticated identity and an explicit tenant. Before retrieval ran, an authorization service worked out which documents that agent and tenant could see. Retrieval searched only those, and access was checked again when source text was read.

Permission metadata traveled with each indexed chunk and its source version. A prompt couldn't widen the scope, and we never counted a model declining to answer as an access control. Index publication was versioned, revocation invalidated cached evidence, and cache keys included tenant, permission scope and corpus version. When authorization was uncertain, the request failed closed.

Evidence all the way through

Retrieval combined keyword and semantic matching over authorized chunks and kept each passage's document, version and boundaries. Documentation owners published approved versions, and withdrawn documents dropped out of retrieval.

The drafting model got a bounded evidence package, with instructions kept apart from document text. Its output was a set of claims tied to specific passages, not a paragraph with a link to the docs homepage. A retrieved document that said to ignore previous instructions was treated as data. When there wasn't enough evidence, the assistant abstained instead of retrying with a bigger prompt.

Checks, then a person

Automated checks confirmed that required citations were present, that claims matched the cited passages, and that the ticket wasn't in an escalation category. They catch obvious failures. They can't certify that an answer is true.

A draft that passed went to the agent with its source passages, their dates and a plain label saying an AI wrote it. The agent edited and approved it in the normal case tool. The assistant had no send button and no access to account actions. High-risk tickets went straight to the manual path with a reason code, and we made that path fast, because editing a bad draft on a hard ticket takes longer than writing the reply from scratch.

Evaluation that tries to break it

We built a held-out evaluation set, de-identified and stratified by product area, permission pattern and escalation category. It deliberately included obsolete documents, contradictory passages, empty retrieval, prompt injection and attempts to pull another tenant's data.

Citation support, correct abstention and how much reviewers had to fix were scored separately. A privacy failure blocked promotion of a new version outright, however much average helpfulness improved. Reviewer corrections went into a controlled feedback process, not straight into training data, and access to the evaluation examples was audited.

The pilot

We started with a small set of documented, low-risk ticket categories. Drafts ran in shadow first, with nothing changing for customers, and then agents opted in. We compared total handling time, review and correction included, against matched manual tickets, and checked that hard tickets weren't dropping out of the comparison.

Generation tokens, concurrency and retries were all capped. If the model was slow or down, the ticket went straight back to the manual path. Rollback restored the previous corpus and prompt configuration, and a model change could never alter the permission policy.

Results

About 60% of the 12,000 monthly tickets were eligible for assisted drafts. On those, handling time went from 18 minutes to 12, with review and editing counted inside the 12. That's 720 hours a month of support capacity.

The running cost covers retrieval, model usage, hosting, evaluation and maintenance. It doesn't include the initial build.

That's recovered time, not a headcount cut or a customer-satisfaction score. The support leads put it toward harder tickets, the backlog and documentation.

Next time I'd write the evaluation set before any retrieval code. The cross-tenant and obsolete-document cases are the ones that decide the design, and it's cheaper to learn that first.

Client name and identifying details are left out.