AI Agent Performance Audit: How I Review My Support Team with AI

I built OT1-Pro to answer customer messages when my team could not. Then one night in March I realized I had a bigger blind spot: I had no idea how my own reps actually performed when I was not watching. I had CSAT scores, I had gut feeling, I had three Slack complaints. None of it told me who closed, who stalled, and who was quietly rude at 11pm.

So I pointed our Deep Analysis engine at our own inbox. Not at leads, not at contacts to enrich — at 4,137 closed conversations handled by six human reps over 45 days. I selected DeepAnalysis::MODE_AGENT_AUDIT, picked the team, and let AgentAuditService do what I never had time to do: read everything.

What came back changed how I run Mondays. Per-rep breakdowns. Tone drift by hour. Seven missed closes worth real money. Two policy violations that could have gotten our Facebook Page flagged. This post is the full playbook for how to audit support team with AI, with the exact metrics I now track.

If you are new to the analysis engine, start with my earlier breakdown of how Deep Analysis processed 3,000 contacts — agent audit mode reuses the same job pipeline but skips the chunk loop entirely, which is why it is fast. And if you have not connected your channels yet, the Meta app verification founder guide walks through the Business verification and App Review steps that unblock the Facebook and Instagram message history you will need.

1. Why I Turned the AI Back on My Own Team

I manage a support operation the way most founders do: I spot-check 10 conversations a week and pretend that is representative. It is not. My best rep on Monday is a different person on Thursday night shift. My worst rep is charming when I am online and short when I am asleep.

The trigger was a lost AED 8,400 deal in Dubai. The customer asked for a quote, asked about onboarding, then went quiet. My rep marked it "not interested." When I finally read the thread two weeks later, the customer had asked "can you send the invoice today?" and the rep replied 9 hours later with "will check tomorrow." That is not "not interested." That is a missed close caused by a slow reply.

I asked myself a painful question: how many more threads looked like that? I could not answer without reading 4,000 threads. No manager can. That is the job I gave to the machine.

An ai agent performance audit is not a CSAT survey. Surveys measure how customers feel after the fact, with 8% response rates skewed toward angry people. An audit measures what reps actually did: every greeting, every delay, every promise, every time they forgot to ask for the sale. It is evidence, not opinion.

2. What Agent Audit Mode Actually Does Differently

OT1-Pro Deep Analysis has several modes. Most of them work in chunks: take 3,000 contacts, split into batches of 100, score each batch, merge results. Agent audit does not work that way, by design.

When you set DeepAnalysis::MODE_AGENT_AUDIT, the job skips the chunk loop and hands the full conversation set for the selected date range straight to AgentAuditService. That service groups messages by rep, builds a per-rep timeline, then scores each rep across four axes: tone consistency, close discipline, reply speed, and policy compliance.

Why skip chunking? Because chunking breaks context. If Rep A handles message 1-3 and Rep B handles message 4-7 in the same thread after a handoff, a chunked scorer might attribute B's save to A. AgentAuditService keeps thread ownership intact and only attributes messages sent by the logged-in rep identity. Handoffs, bot replies, and AI-suggested drafts are tagged separately so humans do not get blamed for the bot's mistakes — or credit for its saves.

Practically, this means a 4,000-conversation audit finishes in one run instead of 40 queued jobs. My 4,137-thread audit completed in 19 minutes. The standard lead-scoring run over the same volume takes over two hours because of the chunk fan-out. If you want the architecture background, the chunked path is documented in the 3,000-contact rerun post linked above.

3. How to Run Your First AI Agent Performance Audit in 6 Steps

You do not need a data team. You need connected history and 20 minutes. Here is the exact sequence I use:

1. Connect history first. Audit is only as good as the threads it can read. Connect Messenger, Instagram, and WhatsApp so page-scoped history flows into one inbox. If Meta shows zero Pages, fix Business-mediated visibility before anything else.
2. Pick a 30 to 45 day window. Seven days is noise. Ninety days mixes in reps who already left. Forty-five days gave me roughly 600-800 threads per rep, enough for stable averages.
3. Create a Deep Analysis job in MODE_AGENT_AUDIT. Select the team, exclude the AI bot sender ID, and include closed plus snoozed conversations. Leave open threads out — they skew close-rate math.
4. Let AgentAuditService build per-rep timelines. Do not filter by tag yet. Let it score everything, then slice by tag (sales vs support) in the report view.
5. Read the per-rep cards before the aggregate. Team averages hide stars and strugglers. Open each rep card: tone chart, reply-time histogram, close-attempt rate, flagged quotes.
6. Export two anonymized examples per finding. You will need them for the coaching session. "Your tone score is 71" changes nothing. "Here are two threads where a warmer close would have kept the deal alive" changes behavior.

Total cost on our plans is one analysis run, not per-message AI credits. That pricing detail matters when you compare vendors — see OT1-Pro pricing versus per-message tools, and my Wati comparison if you are migrating off a per-message WhatsApp tool where bulk reads get expensive fast.

4. The 5 Metrics That Matter: Good vs Red Flag

Most QA scorecards track 25 metrics. Nobody reads them. I track five. Each one maps to money or risk:

That table is the whole coaching meeting. I print one row per rep, highlight the red cells, and attach two quotes. The meeting takes 15 minutes per rep because we argue about evidence, not impressions.

5. Tone Consistency: The Silent Revenue Killer

Tone does not mean "be cheerful." It means the customer gets the same professional rep at 10am and 11pm. My audit showed our team average was 84/100 from 9am to 6pm, then 68/100 from 9pm to 1am. Same people. Different patience.

Anonymized example, Rep C, night shift. Customer: "Salam, the invoice has wrong company name, can you fix before I pay?" Rep reply, 47 minutes later: "send correct name." No greeting, no apology, no confirmation of reissue. Customer never replied. The audit flagged it as tone drop + missed save. When we rewrote it in coaching — "Salam Ahmed, sorry about that, send me the correct name and I will reissue in 10 minutes" — the rep winced. He knew.

Second example, Rep A, same week. Customer angry about a double charge. Rep opened with name, stated the refund amount and date, closed with a direct line. Tone 94. That thread renewed for another year.

My rule now: greeting by name in every thread, one empathy sentence before any process sentence, never close with a single word. Threads scoring 85+ closed at 2.1x the rate of threads under 70 in our data.

6. Missed Closes: Where Deals Go to Die

This is the section that paid for the tool. AgentAuditService flags any sales-intent thread that ends without a close attempt: no price sent, no invoice link, no calendar ask, no "shall I prepare it?" The customer showed intent and the rep let the thread die.

We found 63 missed closes in 45 days across six reps. I sampled 20 manually. Eleven were real opportunities. Four converted on follow-up and produced AED 21,700.

The patterns repeat. Pattern one: the "will check" stall. Customer asks for discount, delivery date, or invoice. Rep says "will check and update you" and never sets a time. Thread dies. Pattern two: the answered question with no next step. Rep answers "yes, we support COD in Sharjah" and stops. No "want me to create the order?" Pattern three: the premature tag. Rep tags "not interested" after one slow reply, which removes the thread from follow-up queues.

Coaching fix is mechanical, not motivational. Every sales-intent thread must end with a question or a link. I added three canned closes reps can send in one tap: invoice request, call booking, and objection ask ("what is holding you back from starting today?"). Close-attempt rate rose from 51% to 74% in five weeks. If you run sales over chat, steal that rule before you hire another rep.

7. Slow Replies and the 4-Minute Rule

Speed covers a multitude of sins. Our data is blunt: threads with first reply under 4 minutes closed at 31%. Threads with first reply over 30 minutes closed at 9%. Everything else held roughly equal. The customer who waits stops believing you will deliver fast.

The audit reply-time histogram showed two distinct problems. Problem one was shift edges: 1pm lunch handover and 9pm night-start both spiked to 28-minute medians. Nobody owned the queue for 20 minutes. Problem two was sticky threads: reps cherry-picked easy chats and left the long English complaint threads sitting. One rep had a 3-minute median on Arabic threads and a 41-minute median on English threads. He was not slow. He was avoiding.

Fixes that worked: a named queue owner for each 30-minute edge window, AI drafts so night reps send in one tap, and a 10-minute breach alert to the shift lead. Four minutes median, 10 minutes breach — that balance held.

For context on cost: speed tooling that charges per AI draft gets punished at scale because night shift generates the most drafts. A flat analysis run plus included drafts is why I could leave drafts on all night without watching a meter. Details are on the pricing page.

8. Policy Violations You Would Never Catch Manually

Tone and speed cost money. Policy costs the channel. Meta and WhatsApp both penalize certain behaviors, and one careless rep can get templates rejected or a Page restricted.

Our audit flagged two hard violations I had never seen in spot-checks. One rep sent a personal bank IBAN over Messenger to "speed up payment" — outside the approved payment link flow, with customer card data then pasted in-thread. Second rep promised "guaranteed approval in 24 hours" for a service that requires third-party review. Both were in threads I would never have opened randomly.

AgentAuditService catches these because it scans every thread against a policy list: payment outside approved links, guarantees, requesting full card numbers or passwords, sharing internal notes externally, and abusive language in either direction. Each hit includes the quote, timestamp, and rep. Soft warnings (mild impatience, missing disclaimer) are separated from hard violations so managers do not treat a missed greeting like a compliance breach.

My process now: hard violations go to same-day review, soft clusters go to weekly coaching. Two hard hits in a quarter triggers retraining before the rep touches sales threads again. Since starting audits, we have had zero repeat IBAN-style incidents in four months. If you run WhatsApp at volume, pair this with the template and opt-in hygiene in the Wati migration notes.

9. The Agency Play: Proving Outsourced Performance to Clients

If you run an agency or outsourced support desk, flip the whole story. Your clients do not believe your "98% satisfaction" slide. An anonymized agent audit report is proof.

One of our agency customers white-labels the per-rep breakdown: rep IDs anonymized to R1-R6, client threads only, tone and speed charts plus three redacted example saves. They send it every month with the invoice. Churn dropped because the conversation changed from "are your agents good?" to "look at close-attempt rate climbing from 58% to 76%." Numbers beat adjectives.

Practical setup for agencies: run MODE_AGENT_AUDIT per client workspace, not blended, so baselines stay fair. Anonymize rep names before export and include two redacted findings per metric: one save, one miss with the rewrite. Clients forward those to their own bosses. That forwarding is your retention.

I also use audits in hiring. New rep probation is 30 days, then their first audit card decides confirmation. No politics. One hire scored 91 tone but 33% close-attempt — perfect for support, wrong for sales. We moved her, and both queues improved. Without the split view we would have fired a good person for being in the wrong seat.

10. What I Do With the Report Every Monday

Reports rot in inboxes. Mine runs the Monday meeting. Total time: 45 minutes for six reps. Here is the agenda:

1. Team slide, 5 minutes. Four numbers vs last week: median first reply, tone average, close-attempt rate, hard violations. Green or red, no speeches.
2. Top save, 5 minutes. Best-handled thread of the week, read aloud, rep named. People repeat what gets applauded.
3. One systemic fix, 15 minutes. Not six individual scoldings. Last month it was shift-edge ownership, then English-thread avoidance, then invoice-link discipline. Fix the system first.
4. Individual cards, 20 minutes. Each rep gets their card async before the meeting with two quotes. Meeting time is only for the one red cell they want to dispute. Most disputes end with "fair" once they read their own words.

Then I re-run the audit monthly, not daily. Daily scoring makes reps game the metric — instant greetings with no substance. Monthly keeps sample sizes honest and coaching calm. Between runs I watch only breach alerts: 10-minute unreplied, hard policy keyword, reopened close-intent thread.

If you have never run one, do it this week. Pull 30 days, run MODE_AGENT_AUDIT, read five threads the report flags worst and five it flags best. You will learn more about your team in one hour than in a quarter of spot-checks. Start free at OT1-Pro registration and point it at history you already own.

Stop losing the leads you already earned

OT1-Pro audits your team, respects Meta's 24h window, pushes leads to your stack via webhooks, and stays up through AI outages with multi-model failover — WhatsApp, Instagram, Messenger, Telegram, and email, in Arabic or English. Free plan, no credit card.

Start free → · Pricing · vs WATI · Talk to the founder on WhatsApp