GEO capture failures: attempts and valid coverage
A timeout does not establish that a brand disappeared. A cut-off answer may show its name yet remain insufficient to check a recommendation. Turning either case into a zero or deleting it from the report mixes assistant behavior with failures in the system observing it.
This guide proposes a record for teams capturing AI answers repeatedly. It links each planned unit to its technical attempts, retained artifacts and acceptance decision. The result explains what was measured and what remained missing, without turning retries into independent observations.
The AI answer variability protocol determines how to repeat questions under comparable conditions. This guide works inside each already planned repetition: recovering a capture where appropriate while preserving its history.
Define the planned unit and acceptance criteria
Assign an identifier to each question, version, configuration, repetition and planned window. That combination identifies a planned unit. A retry retains its identifier; an additional repetition needs a new one and a design rationale, not a repair label.
Before execution, define the evidence required for each metric. Checking ordered position may require the complete list; auditing a citation may require its marker and destination. A capture may support one check but not another. Do not assign a single validity status that hides that difference.
Also define the cutoff, waiting limits and rule for choosing the accepted artifact if several arrive. That rule must depend on completeness, configuration and attempt order, never on brand presence. Keep it with the plan version so another reviewer can reconstruct the decision.
Preserve the unit, attempt and artifact chain
Open one row per attempt with its own identifier and the planned unit's identifier. Record start, end or last observation, time zone, environment and retry reason. Link the preceding attempt where relevant. Do not overwrite its error when saving a later recovery.
Retain available evidence: the original answer, provider-reported status, capture error and associated artifacts. Record only fields exposed by the environment. If no request identifier, completion signal or usage counter is available, mark it unavailable rather than inventing one.
Keep the received file, extracted text and analytical labels separate. A parser failure does not mean the provider failed to answer. If extraction can be repaired from the same authorized artifact, record that new version without requesting another answer or pretending there was another observation.
Classify the result before interpreting brand presence
The following classification is a proposed capture control, not an official provider status catalog. Adapt it to observable signals and retain a reason when a case does not fit.
| Operational status | Evidence to retain | Measurement treatment |
|---|---|---|
| Not started | Planned unit and postponement reason | Remains in the plan without an attributed answer |
| Uncertain delivery | Submitted attempt without a confirmed result | Investigate the same attempt before duplicating it |
| Incomplete capture | Partial artifact and missing element | Assess validity per metric without filling in text |
| Complete capture | Required evidence and recognized configuration | Apply the agreed acceptance rule |
| Confirmed duplicate | Reference to the same event or artifact | Preserve lineage without another numerator unit |
| Outside the window | Observed time and applicable cutoff | Record separately under the closing rule |
A refusal may be a real assistant response rather than a technical failure. Nor should a complete answer be retried merely because it lacks a brand mention. Define how refusals enter each metric and show their count; do not exclude them retrospectively to improve a rate.
Retry without selecting the most favorable answer
Before resubmitting, distinguish a confirmed error from uncertain delivery. If the service supports checking an existing operation, use its authorized mechanism to establish what happened. A client-side timeout alone does not prove that the remote operation never ran.
The Cloud Storage retry strategy distinguishes the error's nature from the operation's idempotency. That documentation concerns Storage; here the distinction serves as a precaution. It does not establish that two requests to a generative model are equivalent or produce identical text.
For APIs exposing these signals, RFC 6585 defines 429 as a request rate limit and permits Retry-After. RFC 9110 describes that waiting field. Follow the service's current documentation and your authorized limits; do not bypass blocks, authentication or usage restrictions.
Document permitted causes, delays, maximum attempts and stopping conditions. Avoid concurrent retries that continue consuming resources after a capture is accepted. If several valid answers belong to one unit, apply the prior rule and retain the rest as technical history, not extra votes or alternatives to choose between.
Reconcile the plan, attempts and accepted captures
At closure, each planned unit needs a status and at most one accepted artifact per metric. Check that attempts link to existing units and that no capture has accidentally been assigned to two units. Preserve earlier historical states even when the final view shows recovery.
Calculate coverage in logical units, not requests: units accepted for a metric within the window divided by units planned for that same metric and window. Keep failed and unstarted planned units in the denominator. Exclusions approved before execution belong to a different scope version.
Show attempts per unit, observed failures and recoveries separately. A technical success rate per request answers a different question from plan coverage. It does not equal billable consumption either: the GEO run budget keeps those units and limits separate.
Do not infer duplication merely because two answers have identical text. They may come from separate repetitions. Use identifiers and provenance to establish that they represent the same event; when that cannot be resolved, leave the case pending and explain its effect on the count.
Examine where missing data is concentrated
Break down planned, accepted and pending units by model, mode, market, language, intent and window. Add dimensions relevant to your design. A total with good coverage may hide an important cell without any usable captures.
Compare conditions for recovered units with those accepted on the first attempt. A later recovery may occur at another time or after a version change. Do not incorporate it as equivalent if it no longer meets the original contract; retain the discontinuity and decide which comparison remains possible.
Do not assume missing cases resemble observed ones. If capture fails more often for one configuration or answer type, the valid set may describe a different part of the plan. Overall coverage does not establish the absence of that bias, and reweighting does not recover an answer never observed.
Display the mention rate alongside its valid count and cell coverage. Interpretation remains limited to that observed set. Do not fill failures with zeros, the last known answer or generated text designed to complete the table.
Close the window without erasing late recoveries
Retain a version of the closure: plan used, cutoff time, acceptance rules and each unit's status. Separate captures awaiting analysis from captures awaiting receipt. Finishing a download does not mean extraction or classification has been validated.
If evidence arrives later, add its arrival time and relationship to the original attempt. Under the approved rule, it remains outside the closure or produces an identified revision. Do not silently change the earlier report or move a late observation into another window as though it ran there.
When retrying is no longer safe, falls outside authorization or reaches the limit, retain the failure. Obtaining an answer at any cost is not required to close the record. Repairing access or configuration needs its own owner and does not authorize bypassing a barrier.
Deliver verifiable coverage before evaluating a drop
The closing package should include planned and accepted units per metric, unresolved states, linked attempts, selected artifacts and exclusion reasons. Add extraction versions, acceptance rules and differences from the plan. Reviewers should be able to trace a figure to its unit without searching an unrelated file collection.
Route capture incidents to the data quality owner. AI visibility drop alerts then consume that coverage control to decide whether the indicator can be evaluated. A measurement problem should not be opened as a loss of brand presence.
This record can accompany measurement managed with Mentio; it does not announce a new retry feature or a recovery guarantee. Its completion criterion is documentary: knowing which capture supports each result, which attempts preceded it and which part of the plan still lacks valid evidence.
If the metric depends on whether the assistant searched, add the observable search-call audit to the artifact. A valid final answer does not establish a complete tool record; acceptance must match the question being measured.
Frequently asked questions
Does a timeout mean my brand did not appear? No. It means the expected result was not obtained within the observed limit. It establishes neither brand absence nor that the remote operation never ran; retain the attempt and its status without turning it into a visibility zero.
Does a retry count as another prompt repetition? Not by itself. A technical retry remains linked to the original planned unit. An additional repetition needs its own unit and design rationale; it should not be introduced as a repair after seeing an unfavorable answer.
Can I use a partial answer if it contains the brand? Only for metrics whose prior acceptance criteria it meets. Seeing the name does not establish that the recommendation, ordered list or citations are complete. Retain what is missing and do not reconstruct absent text.
How do I calculate valid capture coverage? Divide units accepted for a metric and window by units planned for that same metric and window. Count logical units rather than attempts, and show failures, recoveries and pending cases separately.
What should I do with an answer arriving after closure? Retain its arrival and relationship to the original attempt. Apply the agreed rule: leave it outside the closure or issue an identified revision. Do not silently alter the report or present it as an execution from another window.
Want to know if AI mentions your brand?
Discover your visibility in ChatGPT, Claude and Gemini in minutes.
Related articles
How to Measure AI Answer Variability Without Biasing Your Benchmark
Learn how to repeat prompts, control changes and separate signal from noise to build a reliable benchmark of brand visibility in AI answers.
GEO AnalyticsAI Visibility Drop Alerts: How to Separate Noise From an Incident
Design AI visibility alerts with thresholds, persistence, severity, suppression and hysteresis to separate normal noise from real incidents.
GEO AnalyticsGEO measurement budgets: allocate runs without losing comparability
Plan a GEO measurement budget by wave: a stable core, rotation, reserves and spending limits. Record the coverage you give up when the budget is reduced.