NOMOS GEO-QA / English edition

NGQ-060 / Measurement, audit and conformance

How should the question set, sample, language, country, time, model and version scope be selected for a GEO audit?

Short answerChoose the test scope before seeing the results, and make it represent real user intents and material risks.
VERSION
0.3.1
STATUS
founder edition · released
PRIMARY SOURCES
5

Direct answer

Run a GEO audit against a versioned question registry defined in advance. The registry includes entity identity, user intent, question type, risk level, language, country, date, model, product surface and known version. Do not select only examples likely to produce favourable results. Cover basic identity, evidence, comparison, limits, recency, high-impact errors and real user intents in a balanced way. If the scope changes after testing, report the old and new results as separate versions.

In plain language

You cannot judge a class by asking only the three strongest pupils the easiest questions. Decide which subjects and groups should be represented first, then apply the same examination plan to everyone.

Why this matters

Selecting questions after seeing the result can raise or lower the success rate artificially. Representation that looks sound in one language, country or model may fail elsewhere. Without a scope record, two audits cannot be compared.

Do not confuse

  • Exploratory testing seeks unknown error types and need not produce a formal conformance result.
  • Acceptance testing determines whether a version is suitable for use against predefined gates.
  • Regression testing checks whether a change broke earlier correct behaviour.
  • Monitoring testing tracks change in the live system over time.
  • A research sample requires a methodological selection if it will support an inference about a defined population.

What should you do?

  1. State the audit decision, target population and intended generalisation first.
  2. Divide real user intents and critical representation risks into question categories.
  3. For each category, define material strata such as language, country, device or product surface.
  4. Version question wording, order, repeat count and execution conditions before testing.
  5. Record model name, access method, date, time, language setting, country and visible version information.
  6. Report missing or inaccessible strata instead of silently replacing them with other examples.
  7. Open a new question-registry version when the scope changes and do not mix its results with the old version.

How do you audit it?

  • Are the target population and decision purpose of the measurement explicit?
  • Were questions frozen before results were seen?
  • Did easy or favourable questions create selection bias?
  • Are identity, evidence, limits, recency and high-impact errors represented sufficiently?
  • Were language, country, model, surface, date and version conditions recorded?
  • Was a result from one condition generalised to conditions not covered?
  • Were missing, failed or blocked tests removed from the result without disclosure?

Limit

There is no single number of questions or sample size suitable for every GEO audit. Adequacy depends on decision importance, error rate, expected variability, desired precision and the strata to be covered. Do not infer version information that the provider does not expose.

Remember in one sentence

Design the examination before you see the score.

Sources for this record

CITATION RECORD

Muraz, K. (2026). NOMOS GEO-QA: Canonical Question Registry (English Edition, v0.3.1). NobleJackal. https://noblejackal.com/nomos-geo-qa/
© 2026 Kaan MURAZ. All rights reserved.