Back to blogLanes on a red athletics track with white numbers and linesGEO Analytics

    Alternative order: position bias in AI prompts

    2026-11-13·7 min read

    A brand may come first in a comparison because the assistant considers it suitable, because list order influences selection or because answers vary. One run cannot separate those possibilities. Changing the alternatives while changing their positions does not isolate the problem either.

    This guide proposes a test for teams presenting an AI with a closed list of brands or products. It preserves the question and alternatives, balances where each appears and compares selections by identity. The result describes that task's sensitivity to order; it does not measure spontaneous discovery or establish a universal model preference.

    The AI answer variability protocol addresses repetitions under comparable conditions. Here, a different controlled factor is introduced: each alternative's position inside the same prompt.

    Define the selection you want to test

    Specify whether the assistant should choose one alternative, rank all of them or recommend a subset. Fix the audience, need, criteria and constraints. Do not combine answers to different tasks under one selection rate: choosing an option and producing an ordered list yield different outcomes.

    Separate input position from output position. The former records where you presented an alternative; the latter records where the answer placed it. Retain both per entity. Copying an option's visible number into the ranking column confuses the design with the result.

    Research by Pezeshkpour and Hruschka (2024) documents sensitivity to reordered options in multiple-choice tasks. Its models, benchmarks and findings belong to that study. It justifies checking the phenomenon; it does not supply a bias rate for the brands or assistants you measure today.

    Write the conclusion the test could support in advance: stability or sensitivity within the studied set, configurations and window. A candidate list supplied by the team does not represent every alternative a buyer might discover.

    Freeze the question and each alternative's content

    Assign each brand or product a stable internal identifier. Preserve the exact block the assistant will see: name, description, attributes and conditions. Change only its location when producing a variant. Check that every block appears once, with no missing or repeated candidates.

    Keep the question, criteria and requested answer format unchanged. Correcting a typo, shortening a description or adding an advantage while reordering introduces another intervention. If necessary, create a separate version rather than presenting it as the same test.

    Avoid relying on visible letters to identify entities. If the format requires labels, retain the exact label-to-entity map for each variant. Research on selection and option tokens distinguishes problems associated with those identifiers. If labels and order change together, disclose that position alone has not been isolated.

    Balance positions before running the test

    Build a matrix with variants as rows and alternatives as columns. Record each entity's position in each variant and check its frequencies. Giving every candidate a chance to appear first does not establish equivalent exposure in the remaining positions.

    A complete cyclic rotation lets every alternative occupy every position once. It does not cover all neighboring relationships or every possible ordering. If you choose this reduced design, state what it balances and what remains untested instead of calling it exhaustive.

    With all permutations, the number of orderings grows factorially with the number of alternatives. There is no need to claim a reduced sample represents them all. Choose the design according to the decision and resources, retain its rules and check the balance actually achieved.

    Fix variants, repetitions, window and exclusion rules before seeing answers. If one variant receives more valid executions, do not accidentally give it more weight in the total. Document how incomplete blocks will close and which comparisons will remain pending.

    Separate execution order from order inside the prompt

    Running every variant of one type before all others can confound position with time, source availability or a service change. Distribute variants in comparable blocks and fix their sequence before observing results. Keep that sequence alongside each execution record.

    Control session, history, language, mode and available tools within the authorized environment. A conversation retaining earlier answers may introduce additional context; do not treat it as equivalent to a fresh session. Record observable controls and make those the product does not expose explicit.

    Repeat each variant under the agreed design to distinguish between-order differences from variation within the same order. Reordering once and receiving a different selection is insufficient to attribute the change to position. Do not expand only the variant producing a favorable answer.

    Technical failures retain their relationship to the planned unit and do not become independent repetitions. A valid answer choosing a competitor is not an execution failure. Keep refusals, unexpected multiple selections and incomplete captures under their predefined statuses.

    Recover the chosen identity before calculating metrics

    Retain the original answer and link it to the exact submitted variant. Resolve the selected entity through its name or corresponding map. An isolated letter does not identify a brand by itself when the map changes across orderings.

    Record fieldCheck it supports
    Question and versionThe same task across variants
    Variant and position mapWhich entity appeared in each slot
    Configuration, block and repetitionWhich executions can be compared
    Original answerWhat the assistant actually selected or rejected
    Entity and output positionA result not confused with input order
    Evaluation statusValid, ambiguous, failed or pending cases

    Do not force a single selection when the text contains none. Preserve ties, conditions and abstentions under the labeling rules. A conditional recommendation does not become an unconditional win merely to simplify a table.

    Validating the mention and recommendation classifier checks the labels feeding the indicator. Assistant sensitivity to order and an error interpreting its answer are different problems; they must remain separately reviewable.

    Compare results by entity and input position

    For each entity and configuration, show selections and valid executions by the position where it was presented. Include the bases and pending cases. A difference in rates without exposure counts per position may reflect an unbalanced design.

    Also compare distributions across variants and repetitions of the same variant. If results vary widely without changing order, that variation belongs in the interpretation. Do not label every discrepancy between two answers a position effect.

    If calculating a selection change rate, define the pairs to compare in advance and count identity changes rather than letter changes. Report valid and incomplete pair counts. Changing the reference afterward or selecting only certain pairs can change the result.

    For rankings, retain each entity's output position, ties and unranked candidates separately. Report which parts remain stable and which comparisons change. An isolated average rank can hide alternation between very different orderings.

    Aggregated summaries must preserve the question or block sharing conditions as a unit. Do not treat all its variants as independent evidence about the entire market. The AI visibility industry benchmark addresses population and peer comparison; this test does not replace it.

    Do not choose the winning order after measuring

    A sensitivity test loses its purpose if the team publishes only the variant favoring its brand. Retain the planned set of orderings and their results, including conditions under which selection changes or cannot be evaluated.

    The GPS work published in 2026 studies selecting permutations to improve answers on multiple-choice benchmarks. That objective differs from comparably describing brand visibility. Do not present an order-optimization strategy as neutral measurement.

    If the operational prompt is changed to reduce observed sensitivity, evaluate that new version in a separate test. Keep the earlier result as diagnostic evidence. Do not rewrite history or claim that the adjustment removed every system bias.

    Publish the scope alongside the decision

    The closing record should retain the question, candidates, exact content, variants, achieved balance, schedule, repetitions and configuration. Add identity maps, answers, interpretation rules, valid bases and exclusions. Another person should be able to reconstruct what moved and what stayed fixed.

    State the finding within its limits: stable selection across observed orders, sensitivity requiring caution or insufficient evidence to compare. This guide offers no universal approval threshold. The decision depends on usage, coverage and the cost of interpreting presentation-dependent behavior as preference.

    This procedure can accompany measurement with Mentio; it does not announce an automatic permutation generator or product bias correction. It helps avoid confusing a position assigned by the prompt author with a recommendation that would emerge without that list.

    Frequently asked questions

    Can changing the order of alternatives change an AI selection? Research has observed this phenomenon in multiple-choice tasks, but it does not establish a universal rate for every assistant. Check it in the task, configuration and alternative set you need to evaluate.

    Is reversing a list just once enough? Not to isolate stable sensitivity to order. Reversal may leave positions without equivalent exposure, and one answer does not separate the change from ordinary variation. Define balanced variants and repetitions before execution.

    Does a cyclic rotation test every permutation? No. A complete rotation balances each alternative's presence in each position, but it does not cover every ordering or neighboring relationship. Disclose that limit when interpreting the result.

    Should I count the chosen letter or the chosen brand? Count entity identity using each variant's map and retain the original answer. The same letter may represent different entities after reordering; it should not be confused with a recommendation's final position either.

    Does this test measure whether AI discovers my brand spontaneously? No. It measures selection within a supplied list under tested conditions. Studying discovery without candidates requires open questions and separate measurement that does not present that list to the assistant.

    Want to know if AI mentions your brand?

    Discover your visibility in ChatGPT, Claude and Gemini in minutes.

    Related articles