How to validate an AI mention and recommendation classifier

If a classifier mistakes a brand named as a rejected alternative for a recommendation, the dashboard can report an improvement that never occurred. Running more prompts does not fix that error: it supplies more answers to the same labeler.

Validating a classifier means comparing its decisions with answers labeled under a documented human standard that were not used to tune it. This protocol is for the data or analytics team approving those labels before they enter GEO metrics. It applies to rules, supervised models and LLM-based classifiers without assuming they share the same confidence mechanism.

The measurement governance guide establishes who approves changes. Here, the output is the test record used to decide what to automate and what to leave under review.

Define the decision you will test

Fix the evaluation unit. One option is a complete answer plus the target entity, with the question context and sources supplied to the classifier. If you evaluate passages, state their boundaries: do not compare a sentence-level label with one covering the whole answer.

Evaluate mention and recommendation as separate decisions. The labeling manual should specify what counts as the entity's name or alias and what evidence constitutes a recommendation. Define how to handle negations, rejected alternatives, lists without a preference and references to a brand's products. Retain the passage supporting each decision.

A conditional recommendation needs to retain its condition. Citing a page does not, by itself, mean the answer recommends its owner. If an answer is truncated or lacks sufficient context, mark it unevaluable under the agreed rule rather than forcing a negative label.

These definitions belong to your protocol. They are not universal categories shared by every provider or evidence of the model's internal intent.

Build a human reference before measuring the system

Select real answers from the setting in which the classifier will operate. Include relevant languages, surfaces, question types and entities. Separate a sample reflecting routine use from a difficult-case test set; the latter helps find failures, but does not directly estimate their production frequency.

Have two reviewers independently label an agreed portion of the set without seeing the system's prediction. Record the label, supporting passage and reason. Resolve disagreements before closing the reference; do not make the classifier the judge of its own test.

When a disagreement exposes an incomplete rule, revise the manual and revisit affected rows. Preserve the original decisions and their resolution. If a defensible reference remains unavailable, report how many cases are unresolved and exclude them from metrics requiring a known label. That exclusion must remain visible in the report.

Hold back answers that never enter tuning

Separate development, validation and final testing before changing labeler rules or prompts. Development supports construction; validation supports configuration selection; the held-out test estimates the result of the frozen configuration.

Do not distribute copies or very close variants of an answer across these sets. Group by the dependency relevant to your evaluation question. To assess performance on new prompt families, keep each family within one group. To assess future performance, include a time-based separation and explain what it represents.

The scikit-learn cross-validation documentation distinguishes splits for independent observations, groups and time series. Applying that distinction to AI answers requires identifying their dependencies; random row splitting does not remove them.

If you change the classifier after inspecting the test, that set has influenced development. Keep the result as a diagnostic and obtain a fresh held-out evaluation before presenting an independent estimate.

Count errors by label, not just total correct answers

For each binary decision, compare positive and negative predictions with the reference. Keep abstention in a separate column.

Precision is TP / (TP + FP): how many predicted positives were correct. Recall is TP / (TP + FN): how many reference positives were detected. F1 combines them through their harmonic mean. The scikit-learn metrics guide documents these calculations and their variants.

Always report the underlying counts. When a denominator is zero, state that the metric is undefined and disclose the calculation convention; do not present it as perfect performance. High overall accuracy can hide failures on uncommon recommendations.

With abstentions, state which population enters each calculation. You can report precision and recall among decided cases, but should not describe them as performance over the entire set. Separately report reference positives left without an automated decision.

Choose thresholds around the error you can accept

Before inspecting the final test, specify which error matters more for the intended use. A false positive inflates recorded recommendations; a false negative misses them. Set acceptance criteria with the data's users rather than choosing them afterward to pass a version.

If the classifier produces a usable score, compare validation results across thresholds. The scikit-learn threshold-tuning guide separates score estimation from decisions and warns about overfitting when data is reused for tuning.

Do not treat verbal confidence or a number an LLM reports about itself as a calibrated probability. Check its relationship with observed correctness before using it as a criterion. If you only have labels, do not invent a threshold curve: evaluate explicit review rules, such as ambiguous entity identity or incomplete context.

No universal percentage makes every classifier acceptable. Document minimum requirements, exceptions and permitted use before testing. If an important class lacks enough cases for evaluation, the outcome is insufficient evidence, not approval.

Measure the work left by abstention

Allowing the system to abstain may reduce errors among its decisions, but also leaves work unresolved. Report automated coverage as decided cases divided by evaluable cases. Count abstentions by reference label, language and reason as well.

Do not turn abstentions into “no mention” during aggregation. Retain three states: automated decision, pending review and completed human decision. Measure the classifier separately from the full reviewed workflow; crediting reviewer corrections to the system would hide its actual performance.

Compare configurations at similar coverage or present coverage alongside errors. A version deciding only easy cases has not demonstrated a general improvement over one answering nearly all of them.

Inspect segments and the effect on GEO metrics

Break results down by languages, answer types and entities relevant to the approved use. Report the size of each segment and flag those that cannot support firm conclusions. An underrepresented class is not validated by the average of the others.

On the same answers, compare the current and candidate versions against the reference. Also inspect which labels change and how they affect the aggregate indicator. False positives and false negatives can cancel out in the total while still assigning recommendations to the wrong brands.

The GEO KPI guide defines the indicators consuming those labels. The variability guide examines changes across repeated answers. Keep these uncertainty sources separate: variable answers and incorrect classification are different problems.

Close the test with a reproducible decision

The acceptance record should retain:

• Unit, intended population, labels and manual version.
• Development, validation and test sets, with their separation rules.
• Human reference, resolved disagreements and unevaluable cases.
• Classifier version, rules, prompt, model and available thresholds.
• Error matrices, metrics with denominators and coverage by segment.
• Criteria fixed before testing, failures and exceptions.
• Decision: approval for a defined scope, mandatory review or another iteration.

After changing the parser, classifier prompt, model version or intended use, re-evaluate the affected scope. Keep regression tests for known failures and new answers for generalization checks. Preserve original predictions when a review corrects them.

When labels apply to a dialogue, preserve its history too: recommendation measurement in multi-turn conversations distinguishes each answer's state from the brand's trajectory.

Frequently asked questions

Does a brand mention always count as a recommendation?

No. A brand can appear in a negation, comparison or rejected alternative. Evaluate mention and recommendation separately, retaining the passage and conditions that justify the label.

Can I use the same answers to tune and evaluate the classifier?

You can use them for diagnosis during development, but should not present that result as an independent test. Hold back answers that do not enter rule, prompt or threshold selection.

What is the difference between precision and recall?

Precision measures the proportion of predicted positives that are correct. Recall measures the proportion of reference positives detected. Report both with their denominators and explain how abstentions are handled.

Does abstention mean there is no mention?

No. It means the system did not make a decision under the defined rule. Keep it pending or send it for human review rather than silently converting it into a negative label.

What minimum score do I need to approve the classifier?

There is no universal minimum. Set criteria before testing based on error costs, required coverage and intended use. If evidence for an important class is insufficient, restrict approval or collect more data.

Want to know if AI mentions your brand?

Discover your visibility in ChatGPT, Claude and Gemini in minutes.

Analyze your brand for free

Related articles