Risk Review Help Cases AI Agent
Scope: 0-to-1 redesign of the Risk Review support experience — moving from Tasks/Butterfly forms to a structured Help Cases system with an embedded support AI agent, used by project owners across orgs including Instagram, WhatsApp, and Monetization
TL;DR: The brief was to reduce help-case volume. I decided the real problem was unstructured data — project owners were filing free-form requests that neither a human nor an AI could triage efficiently. Restructuring the intake itself took AI answer-confidence from 10% to 67% in the same window that resolution rates rose from 7% to 21%+.
Context
Help cases were generated by project owners across many different orgs — Instagram, WhatsApp, Monetization — as the mechanism for getting human support during the most intractable stages of Risk Review, an inherently complex process. Help cases doubled as a negative signal: a spike of cases on the same topic meant something in the Risk Review UX or content was confusing people. Common, repeat issues burned Privacy Review Program Manager (PRPM) time and attention that could have gone to genuinely novel or anomalous problems instead.
Before this shipped, the intake was a single open text box captioned, essentially, “describe your issue.” PRPMs were fielding dozens of these a week, most of which changed hands repeatedly across cross-functional partners before anyone resolved them — because there was nothing in the case itself to route on.
Reframing
The original brief was to reduce case volume. I decided the actual problem was unstructured data: project owners had a comment box when what they needed was an intake form. It’s the difference between an ER waiting room and a comment card — one captures category, severity, and history before anyone gets triaged; the other just lets you vent. AI needed the ER form, not the comment card. Structured data meant PRPM triage could become automatic. More importantly, it meant AI could start clustering similar problems by shared attributes — the necessary first step before AI could intercept any help cases at all. You can’t automate an answer to a question you can’t parse.
Decisions
- How much to ask of the initial form. Cross-functional partners wanted to backload the AI experience and minimize the fields a project owner had to fill in. I argued the opposite: we needed to raise the bar on the context and quality of incoming requests before AI could do its job at all. Garbage in, garbage out — no amount of agent polish fixes an empty intake field.
- Whether to let project owners skip straight to a human. I pushed back hard on making the AI stage skippable as an “escape hatch.” Skipping it would have defeated the project’s purpose, prevented us from building a real help/resolution corpus, and damaged our self-eval process — more cases, but fewer AI attempts to actually evaluate. We compromised on a cooldown: project owners had to wait for the agent to generate an attempted answer before the skip-to-human option unlocked.
What shipped
- The structured intake form — issue category, a guardrailed description field, a title, and org context, replacing the old single text box
- Conversation design for how the AI agent structured and delivered its stated solution
- Failure-mode ontologies — how the agent behaves differently depending on its confidence level, or when an issue touches material risk
- Off-ramps: null-state messaging when no answer could be found, plus the cooldown button and other UX friction designed to keep project owners on-page long enough for the agent to attempt an answer
The structured intake — validated title and issue-statement fields, plus a guardrailed issue-type taxonomy replacing the old single free-text box.
The cooldown in action — the agent has to attempt an answer before "skip and open ticket" unlocks.
The reviewer side: the AI's attempted answer shown alongside the ticket, with feedback fields that fed straight back into the confidence and resolution-rate numbers below.
Outcomes
- Measured: AI-assisted resolution rate 7% → 21%+ in three months; 12% reduction in Q4 help cases; question coverage rate (how often the agent could confidently attempt an answer at all) 10% → 67%; project owner skip rate (bypassing the agent for a human) 70% → 50% in the first four weeks; a full 0-to-1 transition of the support flow from Tasks/Butterfly forms to a structured Help Cases system with an embedded agent
- Honest gap: the skip rate only cutting to 50% still nags at me. Half of project owners were still routing around the agent after four weeks — the intake was fixed, but trust in the agent itself lagged behind the infrastructure that was supposed to earn it. Structure got AI to a usable answer; it didn’t automatically get people to believe it.
- Downstream: PRPM hours freed for reallocation, since triage became automatic once the data was structured; this project was the first stage for AI interception of help cases more broadly, laying the groundwork for further automation
What this taught me
In any system, human-in-the-loop is zero-sum. A fledgling AI agent still needs a human somewhere in the loop — the real question is where you spend that scarce human attention. We bet on the front of the experience, with structured inputs, rather than leaving structure optional and forcing humans at the back to pick up the slack for it. Garbage in, garbage out — and that’s the instinct I now bring to every AI system I design.