Direct answer
Run a GEO audit against a versioned question registry defined in advance. The registry includes entity identity, user intent, question type, risk level, language, country, date, model, product surface and known version. Do not select only examples likely to produce favourable results. Cover basic identity, evidence, comparison, limits, recency, high-impact errors and real user intents in a balanced way. If the scope changes after testing, report the old and new results as separate versions.
In plain language
You cannot judge a class by asking only the three strongest pupils the easiest questions. Decide which subjects and groups should be represented first, then apply the same examination plan to everyone.
Why this matters
Selecting questions after seeing the result can raise or lower the success rate artificially. Representation that looks sound in one language, country or model may fail elsewhere. Without a scope record, two audits cannot be compared.
Do not confuse
- Exploratory testing seeks unknown error types and need not produce a formal conformance result.
- Acceptance testing determines whether a version is suitable for use against predefined gates.
- Regression testing checks whether a change broke earlier correct behaviour.
- Monitoring testing tracks change in the live system over time.
- A research sample requires a methodological selection if it will support an inference about a defined population.
What should you do?
- State the audit decision, target population and intended generalisation first.
- Divide real user intents and critical representation risks into question categories.
- For each category, define material strata such as language, country, device or product surface.
- Version question wording, order, repeat count and execution conditions before testing.
- Record model name, access method, date, time, language setting, country and visible version information.
- Report missing or inaccessible strata instead of silently replacing them with other examples.
- Open a new question-registry version when the scope changes and do not mix its results with the old version.
How do you audit it?
- Are the target population and decision purpose of the measurement explicit?
- Were questions frozen before results were seen?
- Did easy or favourable questions create selection bias?
- Are identity, evidence, limits, recency and high-impact errors represented sufficiently?
- Were language, country, model, surface, date and version conditions recorded?
- Was a result from one condition generalised to conditions not covered?
- Were missing, failed or blocked tests removed from the result without disclosure?
Limit
There is no single number of questions or sample size suitable for every GEO audit. Adequacy depends on decision importance, error rate, expected variability, desired precision and the strata to be covered. Do not infer version information that the provider does not expose.
Remember in one sentence
Design the examination before you see the score.
Sources for this record
- S38NIST AI 600-1, *Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile*Voluntary standards-oriented institutional guidance
- S44NIST, *AI Risk Management Framework Playbook* and AI RMF CoreVoluntary implementation guidance
- S50NIST AI 100-1, *Artificial Intelligence Risk Management Framework (AI RMF 1.0)*Voluntary standards-oriented institutional framework
- S51NIST/SEMATECH, *e-Handbook of Statistical Methods*Official authority technical handbook
- S52NIST CSWP 31, *Proxy Validation and Verification for Critical AI Systems: A Proxy Design Process*Official authority technical white paper

