Quick answer: Start with a keyword and heuristic classifier that routes reports into four buckets — crash, performance, visual, and gameplay — based on attached artifacts and text signals. Route anything below a confidence threshold to manual triage, then feed those manual decisions back as training data. Most indie teams never need to move past this rule-based system.

Bug reports arrive faster than you can read them. At 200 reports a day, a single triager burns four hours just reading, let alone investigating. Auto-classification flips this around: the common cases get routed, tagged, and deduplicated automatically, and humans only look at the reports where the system is unsure. You do not need a language model or an ML team to build this. You need a dozen good rules, a confidence score, and a loop that learns from corrections.

Pick Categories That Match Your Triage Flow

The categories should mirror how you already work. If your team has separate owners for graphics, networking, and gameplay, classify reports into those buckets. If everyone triages everything, use severity-based categories (crash, blocker, minor) instead. Do not invent taxonomies that look clean on paper but do not change who picks up the report.

For most indie studios, four buckets work: crash (stack traces, process death, unresponsive), performance (framerate, stutter, load time), visual (rendering, UI, animation), and gameplay (logic bugs, balance, progression). Add sub-categories only when you have enough volume in a bucket to justify specializing.

Start with Rules, Not Models

The fastest accuracy gains come from obvious signals. A report with an attached stack trace is almost certainly a crash. A report with a framerate number below 30 is almost certainly performance. A report mentioning “black screen” or “missing texture” is almost certainly visual. Write rules for these patterns first and measure how much ground they cover before worrying about edge cases.

def classify(report):
    signals = {
        "crash": 0,
        "performance": 0,
        "visual": 0,
        "gameplay": 0,
    }

    # Artifact signals (strongest)
    if report.has_stack_trace or report.process_died:
        signals["crash"] += 5
    if report.avg_fps and report.avg_fps < 30:
        signals["performance"] += 4
    if report.has_screenshot and report.gpu_error_count > 0:
        signals["visual"] += 3

    # Keyword signals (weaker)
    text = (report.title + " " + report.body).lower()
    for kw in ["crash", "freeze", "closed", "exception"]:
        if kw in text: signals["crash"] += 1
    for kw in ["lag", "stutter", "fps", "slow"]:
        if kw in text: signals["performance"] += 1
    for kw in ["texture", "glitch", "flicker", "black"]:
        if kw in text: signals["visual"] += 1

    label = max(signals, key=signals.get)
    confidence = signals[label] / max(sum(signals.values()), 1)
    return label, confidence

Artifact signals matter more than text signals. A stack trace is definitive, while the word “crash” in a body can mean anything. Weight your rules accordingly. When both signals agree, confidence is high. When they disagree, the report goes to manual review.

Measure Before You Ship

Export your last 500–1000 triaged reports with their final human labels. Run the classifier against them and compute accuracy per category. If crash accuracy is 95% but gameplay accuracy is 40%, you do not have a classifier problem, you have a gameplay-category problem: the text signals for gameplay are too ambiguous. Either split gameplay into finer sub-categories or accept that gameplay will always go to manual triage.

Track precision and recall separately. Precision is “when the classifier says crash, how often is it right?” Recall is “of all the actual crashes, how many did the classifier catch?” A classifier that labels everything as crash has 100% recall but terrible precision. Aim for both metrics above 85% before auto-routing.

Always Have a Manual Fallback

Set a confidence threshold below which the classifier sends reports to a manual triage queue with its best guess pre-filled. This is the critical part. A classifier that forces a bad guess is worse than no classifier because it hides reports in the wrong bucket. A classifier that admits uncertainty and asks for help is a force multiplier.

if confidence > 0.75:
    report.auto_label = label
    report.assign_to(team_for[label])
else:
    report.suggested_label = label
    report.assign_to("manual_triage")

Store both the prediction and the final human decision. When they disagree, that is the most valuable training signal you have. Review these disagreements weekly. Sometimes the classifier is wrong and you adjust the rules. Sometimes the human is wrong and you adjust the category definitions. Either way, you learn.

Upgrade to ML Only When Rules Plateau

If your rule-based classifier caps out at 80% accuracy and you have thousands of labeled reports, consider a lightweight model. A logistic regression on TF-IDF features, trained on your historical reports, can push accuracy to 90%+ with no deep learning infrastructure. Scikit-learn can train this in ten lines of code, and inference is fast enough to run inline on every incoming report.

Resist the urge to use a large language model for this. LLMs are expensive, slow, and often hallucinate categories. For a four-way classification with structured signals, they are overkill. Use them for freeform tasks like summarizing a long report body, not for classification.

“We built a keyword classifier over a weekend and it cut our triage time in half immediately. The ML version we built six months later was barely better, and it took three weeks. The rules did most of the work.”

Related Issues

For deduplication after classification, read how to set up bug report deduplication for cross-platform crashes. To learn about the triage flow reports feed into, see how to triage bug reports efficiently.

Export your last month of triaged reports today. You already have the training data you need — you just have not looked at it as training data yet.