Back to blogBlack magnifying glass beside a silver laptop on a light marble surfaceGEO Analytics

    Did the assistant search? Audit observable tool calls

    2026-11-15·7 min read

    An answer includes links and looks current. That lets you inspect what it cites, but it does not establish whether it ran a search, which queries it sent or which pages it opened. The sentence "I searched" is not a substitute for a recorded tool event either.

    This guide proposes a record for teams auditing assistant runs: configuration, observable calls, returned sources and answer citations. Its purpose is to let another reviewer check the conclusion against the same artifacts. It cannot recover hidden internal processes or establish every source that influenced an answer. This is a working protocol, not an automated Mentio feature announcement.

    Define the surface and the audit question

    Identify what you can observe: a conversation in an application, an API response or your own system coordinating several tools. Do not put those surfaces in a single evidence category. A provider offering search through its API does not establish that a particular conversation in its application followed the same path.

    Ask a question the record can answer. "Is there a search call in this run's complete record?" differs from "Was our brand's page retrieved?" and "Does the citation support this claim?" Each requires different evidence. Resolving the first does not resolve the others.

    Set the boundary of the record as well. If you receive only the final output from an intermediary service, state that you cannot inspect any tools it may use. Lack of access to its record is not a finding about what it did. Your unit is the observable run, not the model's general behavior.

    Preserve the run before summarizing it

    Assign a local identifier and retain the date with its time zone, the exact question and any authorized context that may affect the result. Record provider, surface, known API or SDK version, declared model and mode. Leave fields unknown when they are not exposed.

    Preserve the submitted configuration and received response before extracting a summary. For an event-based capture, keep the sequence and whatever terminal state you can verify. Final answer text alone can lose calls or annotations stored elsewhere in the artifact. Turning a truncated record into a tidy table must not make it appear complete.

    Separate run, attempt and artifact. A retry may produce a different answer and tool path; do not silently use its events to fill gaps in the first attempt. The capture-failure and coverage record explains how to retain that lineage. Here it ensures that a conclusion about search refers to the correct attempt.

    Do not confuse availability with an observed call

    Record tool availability, submitted controls and returned events separately. Search being enabled describes a possibility or instruction under that integration's contract. On its own, it is not evidence that a call completed.

    For an observed call, preserve its native type, identifier where available, action, exposed arguments, order in the record and status. Do not construct missing queries from words in the answer. Do not count transmission fragments from the same event as independent calls when its identity can be reconstructed.

    Keep "call recorded," "result returned" and "answer completed" separate. If a tool fails and the assistant still answers, retain both facts. The answer does not turn the earlier failure into a successful retrieval or establish that its text came from that failed search.

    Read the contract you actually used

    For Responses, the OpenAI web search documentation distinguishes web_search_call events, actions such as search and url_citation annotations in the message. It also documents include: ["web_search_call.action.sources"] for returning sources. Queries may be absent. Preserve available fields without inventing missing ones.

    The current Google Search grounding guide for Gemini shows Interactions steps named google_search_call with queries, google_search_result with suggestions and citation annotations in text. That is the schema for that integration; do not impose it on another endpoint's response or the consumer application.

    Review the documentation for the captured version before writing the extractor. Retain a dated contract reference or copy and an authorized artifact sample. When the structure changes, create a new extraction version. A field your extractor can no longer read does not establish that the provider stopped searching.

    Link events, sources and citations without inventing joins

    Build relationships that can be reconstructed. Start with run and attempt, then event and exposed query, returned source and citation marker. Use identifiers or explicit links where they exist. If the record supplies only a run-level source list, do not assign each URL to a particular query based on proximity or similar wording.

    RecordWhat to preserveWhat it does not establish alone
    ConfigurationSubmitted tools and controlsThat a call ran
    EventExposed type, identity, action and statusThat every candidate page was read in full
    QueryExposed text and documented relationshipEvery possible internal reformulation
    SourceURL and scope of the field returning itThat it was cited or decisive
    CitationMarker, destination and answer segmentThat the passage supports the claim
    CaptureTerminal state and artifact limitsActivity outside the observed surface

    Retain the original URL alongside any normalized grouping version. Removing parameters or merging hosts may simplify a count but can also conflate different documents. If a relationship between two objects is missing, record "not linkable from this artifact" rather than manufacturing a complete chain.

    For citations, check the segment against the exact text to which its offsets or markers belong. Translation, formatting cleanup or combining blocks may move it. The claim-to-passage support audit comes next: a visible association does not yet establish semantic support.

    Separate observable absence from insufficient capture

    Define states before counting. You can record "search observed," "no search event in an accepted record," "incomplete capture" and "surface without tool details." These are operational labels proposed by this guide, not universal official states.

    The second state requires accepting the artifact as complete for the audited question and integration. Even then, the finding concerns the exposed record: it does not assert that no internal retrieval or activity outside that surface occurred. If the contract cannot support checking absence, do not force that state.

    A citation without a visible event can remain "citation observed; path not verifiable." A call with no query text is still an observed call, but it does not support analyzing query wording. Keep each unknown in its proper field rather than discarding the whole run or filling every field by analogy.

    Compare counts with explicit denominators

    Report planned runs, attempts, accepted artifacts and cases with insufficient observability first. Then count runs with observed search, events and exposed queries as different units. A run may contain several calls; fragments of those calls must not multiply the run count.

    If you report an observed-search proportion among accepted, comparable records, show both counts and how many units were excluded. Do not describe it as the search frequency of all users. Do not compare providers when one exposes calls and another only citations, or when acceptance criteria changed between periods.

    Separate returned sources, unique URLs and citations too. Deduplicating URLs answers a different question from counting appearances. A decline in sources in your table may result from an extractor change, a truncated capture or an observable response change. Check those possibilities before attributing it to assistant behavior.

    Close with a bounded conclusion

    Deliver an artifact index, extractor version, acceptance rules, verified relationships and unresolved cases. The conclusion should state which event was observed, in which run and which part cannot be reconstructed. Do not turn that record into a causal explanation of why the brand appeared or was omitted.

    The crawler access audit checks a different layer: requests to your infrastructure. An assistant search event does not establish a fresh visit to your server, and a server log does not automatically identify the user's run. Join those layers only through a verifiable relationship and retain the remaining uncertainty.

    Keep only material authorized for the review. Do not include credentials, cookies, session signatures or private reasoning in a shared report. Protect the original where necessary and maintain a working copy with documented redactions. The next reviewer needs enough evidence to check the conclusion, not unnecessary access to the entire conversation.

    Frequently asked questions

    Does a link in the answer prove that the assistant searched? It establishes that a link is visible. To substantiate a particular call, you need the event or evidence exposed by that integration; the link alone does not reconstruct its path.

    Can I say there was no search if I cannot see a call? Not as a general conclusion. Distinguish a complete record with no exposed event, an incomplete capture and a surface that does not show tools. Limit the finding to the artifact you can review.

    Can I apply an API test to the same provider's application? Not as though they were the same run. Record surface, configuration and version separately; an API test does not establish the path of a conversation in the application.

    Does a returned source already count as a supported citation? No. Keep the source, any citation to it and the review of the passage supporting the claim separate. They are different relationships and checks.

    What should I do if part of the event record is missing? Mark the capture as incomplete for questions that depend on the missing part. Preserve what was observed and keep any retry as another attempt, without retroactively completing the original.

    Want to know if AI mentions your brand?

    Discover your visibility in ChatGPT, Claude and Gemini in minutes.

    Related articles