Create a Human-Reviewed Support Ticket Triage Workflow

The first support-triage prototype I built assigned every incoming ticket a priority and a category and then posted the result straight into the team’s Slack channel. It looked efficient for about two days. On the third day it labeled a billing outage as “low priority / feature request” and labeled a polite feature suggestion as “urgent / outage.” No one had been asked to confirm either decision. The team lost a few hours of trust that took weeks to rebuild.

That episode is why this workflow exists. We will build the smallest useful version of a support-ticket triage system that:

1. Extracts a structured triage decision from the ticket text

2. Pauses for human review before any routing or notification happens

3. Records both the model’s suggestion and the human’s final decision

No fancy agent loop, no external ticketing API yet—just a clear, reviewable decision step you can later wire into Zendesk, Intercom, or your own queue.

As always:

Build the smallest useful version, then test what breaks.

What You Will Have Working by the End

• A fixed JSON schema for a triage decision

• A model call that produces that decision from raw ticket text

• Validation of the model output

• A human review step that can accept, edit, or reject the suggestion

• A simple audit record of both the model proposal and the final human decision

• Three realistic failure modes and how to notice them early

The output will be something a real support lead can trust enough to act on.

The Triage Schema

We keep the shape deliberately small:

JSON

{
  "category": "billing" | "technical" | "account" | "feature_request" | "other",
  "priority": "low" | "medium" | "high" | "urgent",
  "summary": "string (one or two sentences)",
  "suggested_action": "string",
  "needs_human_review": true
}

In code:

Python

TRIAGE_SCHEMA = {
    "type": "object",
    "properties": {
        "category": {
            "type": "string",
            "enum": ["billing", "technical", "account", "feature_request", "other"]
        },
        "priority": {
            "type": "string",
            "enum": ["low", "medium", "high", "urgent"]
        },
        "summary": {"type": "string"},
        "suggested_action": {"type": "string"},
        "needs_human_review": {"type": "boolean"}
    },
    "required": ["category", "priority", "summary", "suggested_action", "needs_human_review"],
    "additionalProperties": False
}

The needs_human_review field is always present. In this workflow we treat every ticket as requiring review; later you can relax that rule for the lowest-risk categories if you choose.

Extracting the Model Suggestion

We reuse the structured-output pattern from the previous post: clear system instructions, a single user message containing the ticket, and validation before any further action.

Python

import json
import os
from datetime import datetime, timezone
from dotenv import load_dotenv
from anthropic import Anthropic

load_dotenv()
client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))

SYSTEM = """You are a support-ticket triage assistant.
Return ONLY a valid JSON object with these exact keys:
- category: one of billing, technical, account, feature_request, other
- priority: one of low, medium, high, urgent
- summary: one or two sentences describing the customer's issue
- suggested_action: a short recommendation for the support agent
- needs_human_review: always true
Do not wrap the JSON in markdown. Do not add extra keys.
"""

def get_model_triage(ticket_text: str) -> dict:
    response = client.messages.create(
        model="claude-sonnet-5",  # verify current ID in official docs
        max_tokens=400,
        system=SYSTEM,
        messages=[
            {
                "role": "user",
                "content": f"Triage the following support ticket:\n\n{ticket_text}"
            }
        ]
    )
    raw = response.content[0].text.strip()
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as e:
        raise ValueError(f"Model returned invalid JSON: {e}\nRaw: {raw}") from e

    # Minimal validation
    required = ["category", "priority", "summary", "suggested_action", "needs_human_review"]
    for key in required:
        if key not in data:
            raise ValueError(f"Missing key: {key}")

    if data["category"] not in ("billing", "technical", "account", "feature_request", "other"):
        raise ValueError(f"Invalid category: {data['category']}")
    if data["priority"] not in ("low", "medium", "high", "urgent"):
        raise ValueError(f"Invalid priority: {data['priority']}")

    data["needs_human_review"] = True  # force the flag
    return data

If the model output fails validation we raise immediately. In a larger system you would add the retry path shown in the structured-summarizer post; here we keep the happy path short so the review step stays visible.

The Human Review Step

The review step is the heart of the workflow. It presents the model’s suggestion clearly and forces an explicit human decision.

Python

def human_review(ticket_text: str, model_suggestion: dict) -> dict:
    """Present the suggestion and collect the final human decision."""
    print("\n" + "=" * 70)
    print("SUPPORT TICKET TRIAGE – HUMAN REVIEW")
    print("=" * 70)
    print("\n--- Original Ticket ---")
    print(ticket_text.strip())
    print("\n--- Model Suggestion ---")
    print(f"Category:          {model_suggestion['category']}")
    print(f"Priority:          {model_suggestion['priority']}")
    print(f"Summary:           {model_suggestion['summary']}")
    print(f"Suggested action:  {model_suggestion['suggested_action']}")
    print("-" * 70)

    while True:
        choice = input(
            "\nAction: [a]ccept  [e]dit  [r]eject  (type a/e/r): "
        ).strip().lower()
        if choice in ("a", "e", "r"):
            break
        print("Please type a, e, or r.")

    if choice == "a":
        final = dict(model_suggestion)
        final["review_decision"] = "accepted"
        final["reviewed_at"] = datetime.now(timezone.utc).isoformat()
        return final

    if choice == "r":
        reason = input("Reason for rejection (optional): ").strip()
        return {
            "review_decision": "rejected",
            "rejection_reason": reason or "No reason given",
            "model_suggestion": model_suggestion,
            "reviewed_at": datetime.now(timezone.utc).isoformat()
        }

    # Edit path
    print("\nEnter new values (press Enter to keep the model’s value)")
    category = input(f"Category [{model_suggestion['category']}]: ").strip()
    priority = input(f"Priority [{model_suggestion['priority']}]: ").strip()
    summary = input(f"Summary [{model_suggestion['summary']}]: ").strip()
    action = input(f"Suggested action [{model_suggestion['suggested_action']}]: ").strip()

    final = {
        "category": category or model_suggestion["category"],
        "priority": priority or model_suggestion["priority"],
        "summary": summary or model_suggestion["summary"],
        "suggested_action": action or model_suggestion["suggested_action"],
        "needs_human_review": False,  # human has now reviewed
        "review_decision": "edited",
        "model_suggestion": model_suggestion,
        "reviewed_at": datetime.now(timezone.utc).isoformat()
    }
    return final

Three possible outcomes:

• Accepted – the model’s suggestion becomes the final record.

• Edited – the human’s changes become the final record; the original suggestion is preserved for later analysis.

• Rejected – no routing happens; the rejection reason is stored.

In a production system the input() calls would be replaced by a Slack modal, a web form, or a ticket-queue UI. The logic stays identical.

The Full Workflow

Python

def triage_ticket(ticket_text: str) -> dict:
    print("Requesting model triage...")
    suggestion = get_model_triage(ticket_text)
    print("Model suggestion received. Waiting for human review...")
    final_decision = human_review(ticket_text, suggestion)

    # Simple audit log (in real life this goes to a database or object store)
    audit_entry = {
        "ticket_excerpt": ticket_text[:200],
        "model_suggestion": suggestion,
        "final_decision": final_decision,
        "timestamp": datetime.now(timezone.utc).isoformat()
    }
    print("\n--- Audit Entry ---")
    print(json.dumps(audit_entry, indent=2))

    return final_decision


if __name__ == "__main__":
    sample_ticket = """
    Hi, I was charged twice for my subscription this month.
    The invoice shows two identical payments of $49 on the 3rd.
    Can someone refund the duplicate charge? Order ID 88921.
    """
    result = triage_ticket(sample_ticket)
    print("\n=== Final Decision ===")
    print(json.dumps(result, indent=2))

When you run it you will see the model’s suggestion, be asked to accept/edit/reject, and then receive a complete audit record.

Three Failure Modes Worth Testing

1. Model Mis-prioritizes a Real Outage

Feed a ticket that clearly describes a service outage and confirm the model’s priority. If it returns “low” or “medium,” the human review step is the only thing standing between that mistake and an incorrect routing decision. Log every case where the human changes the priority; those cases become future evaluation examples.

2. Reviewer Rubber-Stamps Everything

If the review UI is slow or noisy, humans start accepting every suggestion without reading. Keep the presentation short, show the original ticket and the suggestion side-by-side, and periodically sample accepted tickets for quality. Approval fatigue is a process failure, not a model failure.

3. Invalid Model Output Crashes the Workflow

Force the model to return malformed JSON (for example by temporarily weakening the system prompt). Confirm the workflow raises a clear error before the human review step is shown. A triage system that shows a broken suggestion to a reviewer is worse than one that fails loudly and asks for a retry.

Practical Rules That Keep the Workflow Safe

• Never route, notify, or close a ticket on the model’s suggestion alone.

• Always store both the model proposal and the final human decision. The difference between them is the most valuable training signal you will get.

• Keep the schema small. Every extra field is another place the model can be wrong and another field the reviewer must examine.

• Make rejection a first-class outcome, not an afterthought.

• Measure how often humans edit or reject the model. A rising edit rate is an early warning that the prompt or the schema needs work.

What This Workflow Does Not Do Yet

It does not:

• Connect to a real ticketing system (Zendesk, Intercom, etc.)

• Send Slack or email notifications after approval

• Automatically assign the ticket to a specific agent or queue

• Handle attachments, screenshots, or multi-turn customer conversations

• Run as a background worker with retries and dead-letter handling

Those pieces belong to later production posts. The goal today is a single, reviewable decision step that prevents the model from acting alone.

Verification Checklist

1. Run the sample billing ticket and confirm a structured suggestion appears.

2. Accept the suggestion and confirm the audit record contains both the model output and the “accepted” decision.

3. Edit the priority and confirm the final record shows the human’s value while preserving the original suggestion.

4. Reject the suggestion and confirm no routing side effects occur and the rejection reason is stored.

5. Temporarily break the model output and confirm the workflow fails before the review step is presented.

If all five steps pass, the human-reviewed triage workflow is working.

Update Note

Model identifiers and structured-output helpers continue to change. Always verify the current recommended model ID and the preferred way to request JSON in the official Anthropic documentation. The human-review pattern itself is independent of those details; keep the pause-and-confirm step even when the platform offers stronger output guarantees.

Looking Ahead

The next post builds a small internal research assistant that tracks its sources. Once you can reliably extract structured decisions and keep a human in the loop, adding source citations and light retrieval becomes a natural extension of the same production discipline.

Keep the triage script. When you later connect it to a real ticket queue, you will already have the review gate and the audit trail that keep a single bad model suggestion from becoming a customer-facing mistake.

Build the smallest useful version, then test what breaks.

— Caleb Fletcher

Boise, Idaho