By The D2C Agent Desk team · Updated October 2026

Gorgias AI Agent audit checklist: find out why automation has stalled

If your AI Agent went live months ago and the automation rate has flattened, an audit usually turns up the same handful of fixable causes. Here is the checklist we use.

An AI support agent that plateaus is rarely "not smart enough". Far more often, it is missing a fact, following an outdated rule, or handing over topics it could safely own, or the reverse: answering topics it should not touch. A good audit separates those cases with evidence from your own tickets rather than opinions.

Plan on a read-only first pass. You should not change anything in Gorgias until you have the findings written down, so every change can be reviewed and signed off before it goes live.

Step 0: Agree the goal and the guardrails

• What does "better" mean this quarter: higher automation rate, faster first response, fewer reopens, or better CSAT on AI-handled tickets?
• Which topics are off-limits for AI no matter what (for example safety, allergies, chargebacks)?
• Who signs off on guidance or rule changes, and how quickly?

Write these down. They stop the audit drifting into "automate everything".

Step 1: Pull a representative ticket sample

Take two to four weeks of tickets, split into three buckets:

1. Tickets AI Agent closed on its own
2. Tickets AI Agent handed over
3. Tickets AI Agent never touched (other channels, excluded topics, rules)

Sample 30–50 from each bucket. If you have a seasonal spike (a launch, BFCM, a stock-out), note it, because it skews the mix.

Step 2: Read the closed tickets for quality, not just volume

For each AI-closed ticket, check:

• Was the answer factually right against your current policies?
• Did it actually resolve the request, or did the customer just stop replying?
• Did the customer reopen or write in again within a few days?
• Was the tone right for the situation (a lost parcel is not a moment for exclamation marks)?

Every wrong answer goes into a log with the knowledge source it came from. In our experience most wrong answers trace back to a duplicated or stale page, not to the model.

Step 3: Categorise every handover

Gorgias hands over for built-in signals (frustration, explicit requests for a human, sensitive topics), for your configured handover topics, and when it cannot produce a reliable answer. Tag each handed-over ticket in your sample with one of these:

The "correct handover" bucket matters. A healthy audit does not try to push it to zero.

Step 4: Review the knowledge base

• One canonical page per policy (shipping, returns, exchanges, warranty, subscriptions).
• Explicit numbers: processing time, cut-offs, return window, regions served.
• No contradictions between help center, website pages and uploaded documents.
• Product facts available in text (materials, ingredients, sizing), not only in images.
• Seasonal notices removed once the season is over.

Step 5: Review guidance and skills

• Does each guidance cover exactly one topic, as Gorgias recommends?
• Is each one still true? Look for old promotions, old carriers and old return windows.
• Do skills spell out when to stop and hand over?
• Is the knowledge toggle on each skill set deliberately? With it off, the agent works only from the skill's instructions.

Step 6: Review actions and permissions

List every action AI Agent can run (for example cancel order, edit address, send return portal link, subscription changes). For each one, confirm the conditions under which it runs, what happens if it is wrong, and whether it has been tested on edge cases such as partially fulfilled orders, orders with multiple shipments, or subscription orders. Remove or pause anything nobody can explain.

Step 7: Re-test in the Playground

Turn the handed-over and wrong-answer tickets from your sample into a regression test set. Run them through the Playground, which Gorgias documents as free and isolated from real customers and reporting. Use the reasoning view to confirm the agent is pulling from the source you expect. Keep this test set: you will rerun it after every significant change.

Step 8: Write the change list as drafts

Present findings as a short, prioritised list, each with the evidence and the proposed change:

1. Quick knowledge fixes (low risk, high impact)
2. Guidance rewrites and new single-topic guidance
3. Handover topic changes (narrowing or adding)
4. Candidate actions/skills, with test results

Nothing goes live until the owner signs off. Ship in small batches so you can see which change moved which number.

Step 9: Set up weekly reporting

An audit is a snapshot; the weekly report keeps it honest. Track:

• Automation rate, overall and per topic
• Top five handover reasons with one example each
• Reopen rate on AI-closed tickets
• CSAT on AI-handled versus human-handled tickets
• Wrong-answer log: new entries and fixes

Our weekly KPI digest guide shows a format a founder can read in two minutes.

Step 10: Write the runbook

Finish with a one-page runbook: who reviews handovers, where to update policy facts, how to add a guidance, how to rerun the test set, and how to pause the agent quickly if something goes wrong. When the person who set it up leaves, the runbook is what keeps it working.

What an audit will not do

It will not guarantee a specific automation rate. Ticket mix, catalog complexity and policy clarity vary a lot between brands. What it should give you is a clear list of why tickets are handed over, which of those are fixable, and evidence for each change. If you are starting from scratch rather than auditing, begin with our Gorgias AI Agent setup tips.

— The D2C Agent Desk team