A Single Response Is Not a GEO Score

Chapter Boundary

This section only answers the following question:

Why is a single generative system response not the GEO score, and what judgements can be inferred from a single response?

This chapter:

• the sampling quotas of countries,
• how languages will be weighted,
• how the Truth Pack will be set up,
• how adjudicators will evaluate atomic claims,
• the final NOMOS score formulas,
• the appropriateness and badge decision

This chapter does not define these matters in detail; later chapters address them separately. Chapter 1 has a more fundamental task:

To establish the epistemic boundary between observation and generalisation.

Any sampling, scoring, and conformity studies conducted before establishing this boundary will be built on a wrong foundation.

PROVISION STATUSES IN THIS SECTION

Not all sentences in this book have the same normative force. The following statuses are used:

NORMATIVE — NORMATIVE

A provision considered mandatory for conformity and methodological integrity. Example: A single response cannot be used as the general GEO score.

CANDIDATE — CANDIDATE PROVISION

A recommended method to be finalised after pilot, external application, or statistical validation. Example: 1,000 valid observations per AI product, per main measurement wave, are recommended as the assumed minimum sample.

HYPOTHESIS

It is an expectation that needs to be tested with data. Example: There may be a difference in material representation between a controlled clean panel and a natural user panel.

SYNTHETIC

It is a method representation that does not represent a real user, a real AI output, or real company performance.

OPEN QUESTION

It is a topic that has not yet been concluded due to insufficient evidence or governance decision. These statuses cannot silently transform into one another. A candidate threshold does not become a proven normative fact simply because it is written in the book. A synthetic representation does not turn into a real-world observation. A hypothesis cannot be published as a result without being measured.

NOMOS Challenge

You asked an artificial intelligence system a question about your brand. The system gave a correct answer. You took a screenshot. You included it in your presentation. Then you said the following sentence: “Artificial intelligence knows us correctly.” What exactly did you measure? The answer that all users would see? The typical behaviour of the system?

A global representative? Accuracy in all languages? Stability over time? Or only:

The single output that a specific user received, on a specific account, on a specific AI product and user surface, in response to a specific prompt, at a specific time?

A single answer is not worthless. A single answer can be true. A single answer can be important. A single answer can reveal a serious mistake. A single answer can show the real experience of a particular person. However, a single answer cannot be proof of a judgement greater than it can bear.

A single response can show the following:

This output has been produced at least once under recorded conditions.

A single response alone cannot show the following:

• That most users saw the same answer
• That the same result occurred in other countries
• That the same accuracy is maintained in other languages
• That the same answer was produced in other accounts and plans
• That the answer is stable over time
• That other AI products produced the same representation
• That the GEO intervention caused this result
• That the answer affects user behaviour
• That the answer generates lead, sales, or sustainable value
• That the overall GEO score of the entity is high

The first premise of this section is:

A response is an observation. / It is not a distribution of observations. / A distribution is not causality. / Causality alone is not sustainable value.

1. PURPOSE OF THE CHAPTER

This section aims to eliminate the smallest but most common inference error in GEO measurement:

Counting a single output of a generative system as general system behaviour, population experience, brand visibility, or GEO score.

This section normalises the following distinctions:

• Single output versus population outcome
• Existence of an event versus prevalence of an event
• Observation versus sample
• User versus session
• Repetition versus independent observation
• AI model versus AI product presented to the user
• Raw response versus unit of analysis
• Immediate result versus temporal stability
• Screenshot versus full evidence package
• System rejection versus invalid data collection
• Critical singular event and prevalence rate
• Synthetic record and real user observation
• Measured rate and conformity decision

Before these distinctions are established:

• GEO-1000 sample,
• country and language weights,
• atomic claim scoring,
• NOMOS score,
• conformity decision

cannot be established.

2. CENTRAL NORMATIVE PROVISION

The GEO score of an entity in generative systems cannot be generated from a single response, a single screenshot, a single user, a single account, a single session, a single language, a single country, or a single point in time measurement. A population-representative claim requires at least the following:

• Predefined audit object
• Predefined target population
• Specified sampling design
• Locked prompt or prompt family
• Defined AI product and user surface
• Multiple valid observations
• Known dependency structure
• Country, language, and time records
• Unaltered raw responses
• Evidence packages
• Predefined evaluation rules
• Uncertainty statement
• Accountable human responsibility

The presence of these elements does not automatically mean that the measurement is correct. However, their absence may make the generalisation to a large population untenable.

3. FIVE SEPARATE UNITS OF MEASUREMENT

A common fundamental error in GEO auditing is confusing the different units of measurement. The following five units should be clearly distinguished.

3.1. Observation Unit

The observation unit is the single generative system output obtained under specific conditions. An observation can be defined by the following combination:

O_i = (E_i, A_i, U_i, C_i, L_i, S_i, P_i, T_i, R_i, K_i)

Here:

• Ei: the audited entity
• Ai: AI product and user surface
• UI: anonymous contributor or user unit
• Ci: country or geographical stratum
• Li: language and locale
• Si: session and personalisation condition
• Pi: prompt ID and version
• Ti: time and measurement wave
• Ri: raw response
• Ki: proof and verification record

If one of these elements changes materially, a new observation may occur. For example, the same user:

• if it asks the same prompt in another AI product,
• If he asks in another language,
• If he asks in another measurement wave,
• If he asks separately in the personalised and clean session,

Separate observations can be produced. These observations are not automatically considered independent from each other because they come from the same user.

3.2. Sampling Unit

The sampling unit is the person, account, session, or other access unit selected from the target population. In most GEO-1000 applications, the sampling unit can be an appropriate real user. However, according to the research design:

• user,
• user–product pair,
• account,
• session,
• research panel member

It can be used as a unit of election. The failure to clearly define the sampling unit can lead to the following types of miscounts:

• Counting one person's ten repetitions as ten people
• Counting a joint account as multiple independent users
• To evaluate the results of the same user's ten AI products as if they were ten independent population observations
• Ignoring the clustering in a single research panel

3.3. Analysis Unit

The unit of analysis is the entity to which accuracy or representativeness assessment is applied. This unit:

• exact answer,
• separate paragraph within the answer,
• material proposition,
• atomic claim,
• citation,
• recommendation issue

It is possible. A user sees a single answer. However, that answer can simultaneously contain:

• five correct claims,
• two unsupported claims,
• one outdated claim,
• one critical error

Therefore, the answer seen by the participant does not have to be a single, indivisible unit of truth/falsehood in the analysis. The transition status of the full answer and the truth statuses of the atomic claims within it can be maintained separately. The atomic claim method will be defined in a later chapter of the book.

3.4. Universe of Inference

The inference universe is the target population to which the measurement result is intended to be generalised. This universe cannot be silently accepted as "everyone in the world." For a specific AI product, the target universe can be limited by conditions such as:

• Having official and legitimate access to the product
• Meeting the appropriate age requirement
• Being able to use the product in the relevant language
• Being able to access the specified web or mobile interface
• Being able to be active during the specified measurement period

Even for the same AI product:

• controlled clean panel,
• natural user panel,
• free plan,
• paid plan,
• web,
• mobile

can create different inference universes. The published rate is incomplete if it is not specified to which universe a sample generalises.

3.5. Estimated Quantity — Estimand

A study does not just collect data. It tries to estimate a specific quantity. The primary estimand of GEO-1000 can be expressed in its most general form as follows:

The probability that an appropriate user from the defined target population, under the specified measurement wave, on a particular AI product and user surface, sees a representation that meets the acceptance criterion when using a locked prompt under defined session conditions.

Mathematically:

θ_{E,A,P,S,W,T} = Pr(Y_i = 1 | U_i ∈ T, E, A, P, S, W)

Here:

• E: audited entity
• A: AI product and user surface
• P: prompt version
• S: session condition
• W: measurement wave
• T: target population

Yi=1: the response meets the transition criterion that will be defined later

This definition sets an important boundary:

The GEO score is not a model's abstract and timeless 'knowledge'; it is a prediction of the likelihood of observed representations under defined user conditions.

The final transition rule and score formula will be arranged in the later sections of the book.

4. THE LIMITATION OF THE SENTENCE “THIS IS HOW THE MODEL RESPONDED”

Users often do not access a bare base model directly. The result visible to the user may consist of a combination of the following components:

• Base or guiding model
• Product interface
• Retrieval or web access
• Citation layer
• Security and policy layer
• System instructions
• Account plan
• Personalisation
• Memory
• Previous conversation
• Language and locale
• Country
• Time
• Provider's product updates

For this reason, the preferred expression in the main inspection conducted through the user surface is as follows:

"The specified AI product and the following output were observed on the user surface."

The following expression may be broader without the necessary technical control: "The model thinks this way." The model name appearing within a product can be recorded. However, the result should be evaluated as the behaviour of the entire user surface, not just the model weights.

5. WHAT IS A VALID OBSERVATION?

The presence of a response does not automatically mean it is a valid observation that can be entered into the GEO measurement.

5.1. Minimum Validity Conditions

An observation must contain at least the following elements according to the relevant protocol:

• Presence of correct audit
• Assigned AI product and user surface
• Correct prompt ID and version
• Correct language and locale
• Defined session condition
• Recorded country or geographic layer
• UTC timestamp and measurement wave
• Complete raw response
• First unselected suitable output
• Required screenshot or equivalent evidence
• Anonymous observation ID
• Protocol deviation record

Their detailed technical schema will be defined in the later NOMOS Capture section.

5.2. Observation Statuses

Each record must be marked with one of the following statuses:

VALID — VALID

Meets the required conditions of the protocol.

CONDITIONALLY VALID — CONDITIONALLY VALID

There is a limited deficiency; which analyses it can be used for is clearly specified.

INVALID — INVALID

Cannot be used in the main estimation or score calculation due to protocol violation.

UNKNOWN — UNKNOWN

There is not enough record for a validity decision. An UNKNOWN observation cannot be silently accepted as VALID.

5.3. The Difference Between a Valid System Result and Invalid Data Collection

This distinction is mandatory. An AI product:

• may refuse to respond,
• may display an error message,
• may stop without generating a response,
• may experience a connection issue,

may say it cannot complete the task. If the participant applied the entire protocol correctly, these can be valid system outcomes. They cannot be quietly removed from the dataset. In contrast:

• the participant truncating the response,
• changing the prompt,
• regenerating to select the best answer,
• editing the screenshot,
• using the wrong AI product

data collection can create invalidity. The first is the actual behaviour of the system. The second is the disruption of the measurement process. The two cannot be placed in the same category.

6. WHAT CAN A SINGLE RESPONSE PROVE?

A single observation is limited. But it is not meaningless.

6.1. That a Specific Output Occurred at Least Once

A valid record can support this sentence: “On the specified AI product and user surface, under the specified time and conditions, this output was observed at least once.” This sentence shows the existence of the event. It does not show its prevalence.

6.2. A Specific User’s Actual Exposure

A user is mistaken for a:

• company identity,
• health information,
• legal guidance,
• price,
• licence,
• advice

if seen, this is a real user experience. Even if it does not represent the entire population, it is material from the perspective of the affected user.

6.3. The Possibility of a Specific Error

A single output may show: “This error may occur under these system conditions.” However, it does not show: “This error is typical.”

6.4. Generating a New Research Hypothesis

A single error can create a question that will be tested on a broader scale. Example: “Did this identity mix-up occur in a single session only, or is there a systematic pattern in the same language and country?”

6.5. Initiating a Critical Incident Review

Single response:

• predictable serious human harm,
• fake professional authority,
• incorrect emergency medical guidance,
• material legal error,
• serious safety advice,
• serious identity confusion with another entity

may be sufficient to initiate an incident review. Initiating a critical incident investigation is not the same process as announcing a population ratio.

6.6. Refuting the "This Never Happened" Claim

Suppose an organisation says, ‘The system has never assigned us to the wrong country.’ A single verified counter-observation can refute that absolute claim. A single observation does not support ‘It happens all the time’, but it can refute ‘It has never happened.’

7. WHAT CAN A SINGLE RESPONSE NOT PROVE?

7.1. Typical System Behaviour

A single response is not sufficient to conclude: 'The system usually responds this way.'

7.2. Population Prevalence

A single response does not support the result: '95% of users see the same representation.'

7.3. Temporal Stability

Today's response:

• tomorrow,
• a week later,
• After the product update

it may not remain the same. It is not a one-wave stability measurement.

7.4. Cross-Country Validity

A response from a user in Turkey does not indicate what users in China, Germany, or San Marino would see.

7.5. Cross-Language Validity

An English output:

• Turkish,
• Japanese,
• Arabic,
• German

do not represent the accuracy of outputs.

7.6. Cross-AI Product Validity

The result in one AI product cannot be generalised to other AI products. Different products and interfaces of the same provider may also behave differently.

7.7. Intervention Effect

A single positive output after a GEO change does not prove the statement: “This result was created by our intervention.” Baseline, comparison, and alternative explanations are required.

7.8. Recommendation Stability

The presence of a user viewing an option is not sufficient to conclude: “The system now recommends this brand.”

7.9. Commercial Impact

A positive response:

• indicates that the user saw the response,
• researched the brand,
• that you communicated,
• that you purchased,
• that you were satisfied

does not show on its own.

7.10. GEO Score

A single response does not estimate the representative distribution of the defined target population. Therefore, it cannot produce an overall GEO score.

8. THE EXISTENCE OF THE EVENT AND THE PREVALENCE OF THE EVENT

This is one of the most important distinctions in this section.

8.1. Existence of the Event

Answers the question: "Did this representation or error occur at least once?" A single valid observation may be sufficient. Representation:

I(E) = 1 (if the event was observed at least once); otherwise I(E) = 0

The 0 here does not mean: "The event is impossible." It only indicates that it was not observed in the current measurement.

8.2. Prevalence of the Event

It answers the question: "How often does the event occur in the defined target population?" In a simple sample, the raw rate can be calculated as:

p̂ = number of valid records where the event was observed / total number of valid records

However, in real GEO-1000 studies:

• population weights,
• country and language strata,
• matched users,
• panel clusters,
• nonresponse,
• invalid records,
• design effect

should be taken into account. Therefore, the raw rate is not always the final population estimate.

8.3. Incidence and Prevalence Cannot Be Converted Into Each Other

An event may have been observed once but be very rare. An event may be common but not observed at all in a small pilot. The following two sentences are different:

• "This error was observed once."
• "This error was observed in 12% of users."

The second sentence requires a defined denominator and sample.

9. THE STATISTICAL APPEARANCE OF THE SINGLE OBSERVATION BIAS

We asked a question to an AI product once. The answer met the success criterion. Raw rate:

p̂ = 1/1 = 1 = 100%

It appears to be 100 per cent. But that 100 per cent does not mean that every member of the target population will see the same result. For illustration only, the table below shows approximate 95 per cent Wilson confidence intervals under independent Bernoulli observations and unweighted simple random sampling: [K09]

This table is synthetic and illustrative. It does not prove:

• That these intervals are valid for all sampling designs of GEO-1000
• That 1,000 observations are sufficient for all language and country subgroups
• clustering and weighting are ineffective
• that 95 per cent raw accuracy automatically means appropriateness

What it actually shows is:

As the number of observations decreases, the uncertainty about the population proportion increases.

100 per cent seen in one observation can be compatible with a very wide possible population range.

10. WHY “1,000” IS NOT A MAGIC NUMBER?

CANDIDATE BASE RULE — CANDIDATE FOUNDATIONAL RULE

GEO-1000 carries this founding suggestion:

For each monitored AI product and each main measurement wave, a minimum of 1,000 valid user observations is targeted by default.

This provision is not yet the final scientific minimum. It will be re-evaluated based on pilot and external application results.

10.1. Why Is It a Strong Start?

Under the simple random sampling assumption, when the true rate is around 50%, the theoretical 95% margin of error for 1,000 observations is approximately ±3.1 points. This can provide a meaningful starting precision at the overall product level.

10.2. Why Doesn't It Solve Every Problem?

1,000 observations:

• to dozens of countries,
• to many languages,
• to free and paid plans,
• to web and mobile surfaces,
• to controlled and natural panels

When divided, the subcells shrink. For example, if there are only 50 observations for each of the 20 languages, language-based uncertainty can be much higher than the overall model score.

10.3. Design Effect

Observations:

• from the same research panel,
• from the same country,
• from the same user,
• from a matched design

are present, the simple independent sample assumption may not hold. The actual uncertainty can be wider.

10.4. The Approach of NOMOS

1,000 observations serve as:

• the founding reference point of the protocol,
• default starting minimum at the model level,
• candidate threshold to be calibrated with the pilot

NOMOS does not claim that 1,000 observations are automatically sufficient for every country, language and condition of use.

11. INDEPENDENT OBSERVATION IS NOT THE SAME AS REPEATED OBSERVATION

A user can ask the same question ten times. This process can be valuable. However, it does not produce ten independent users.

11.1. Same User–Same Session Repeat

Can measure:

• answer renewal variance,
• within-session stability,

probability of alternative answers. It does not measure population prevalence on its own.

11.2. Same User–New Session Repeat

It may reduce the previous chat context. However:

• same account,
• same plan,
• same country,
• same personalisation,
• same user

may continue to create statistical dependence between observations.

11.3. A Single User Testing Multiple AI Products

This design can help with cross-model matched comparison. However, observations are dependent at the user level. This dependence should be preserved in recording and analysis.

11.4. Different Users Using the Same Account

Different people may exist. However:

• account history,
• personalisation,
• plan,
• memory

can be shared. These records cannot be considered independent automatically.

11.5. Normative Independence Provision

Observations should be designed to be as independent as possible; known user, account, session, panel, country, device, and repeated dependencies should be recorded and taken into account in the analysis.

Independence should not be reduced to a binary:

• yes,
• no

Where necessary, use one of the following dependency statuses:

INDEPENDENT

PAIRED

CLUSTERED

REPEATED_WITHIN_USER

DEPENDENT

UNKNOWN

12. FIRST SUITABLE OUTPUT PRINCIPLE

In the main population panel, the participant's task is not to find the most positive answer. The participant must record the first completed suitable output generated under the predefined condition. The participant:

• cannot refresh the answer,
• cannot request a new answer,
• cannot choose the positive one,
• cannot only trim the good part,

cannot rewrite the answer. Reproduction behaviour can also be investigated separately. However: "We asked the same question five times and selected the best answer." is not the main population score. This is the selected best output test. It should be reported with a separate name and method.

13. SINGLE CRITICAL EVENT EXCEPTION

A single answer cannot determine the population score. However, this does not mean that a single answer can never have a significant impact on a decision. NOMOS establishes two separate assessment pathways.

13.1. Population Representation Pathway

It asks the question: “How frequently does this outcome occur in the target population?” Requires a large and valid sample.

13.2. Critical Event Pathway

It asks the question: “How large is the foreseeable harm when this outcome occurs once?” A single confirmed outcome can trigger a review. For example:

• Assigning a non-existent health licence
• Making a dangerous emergency health referral
• Reporting incorrect legal authority
• Removing the security boundary
• Confusing a company with another legal entity
• Repeating a manipulative instruction hidden from people as if it were advice

It can be a singular Critical event candidate.

13.3. Separation of Score and Conformity Decision

A Critical event can generally be a single observation. For example:

1/1000=0.1%

This rate may seem small. However, if the potential harm of the event is severe, it may create an irreparable gap in the conformity decision. Therefore:

Population score measures the frequency of the event; Critical event review measures the impact of the event on harm and integrity.

High average accuracy cannot automatically eliminate heavy singular error. The final conditions for critical classification will be defined in later chapters of the book.

14. OBSERVATIONAL CLAIM LADDER

In this section, only the first four observation levels are normatively separated. At higher levels:

• population-weighted score,
• multi-model score,
• time stability,
• intervention effect,
• sustainable value

will be defined in subsequent sections.

O-0 — NOT TESTED

The relevant AI product or condition has not been measured. Correct status:

NOT TESTED

Indefensible sentence: "It is appropriate because the problem was not observed."

O-1 — SINGLE EVENT OBSERVATION

There is one valid outcome. Defensible sentence: "Under the specified conditions, this outcome was observed once." Indefensible sentence: "The system represents existence this way."

O-2 — USER OR SESSION REPEAT

Multiple outcomes exist for the same user, account, or session family. Defensible sentence: "Under the specified user and session conditions, this outcome occurred in seven out of ten repetitions." Indefensible sentence: "Seven out of ten independent users saw this outcome."

O-3 — MULTI-USER SINGLE CELL MEASUREMENT

The same:

• AI product,
• user surface,
• country,
• language,
• panel condition,
• prompt version,
• measurement wave

contains multiple suitable user observations. Defensible sentence: “78% of the 200 valid observations in the Turkish/Turkey controlled cell met the specified transition criterion.” This result cannot be automatically generalised to other countries, languages, or user surfaces.

O-4 — MULTI-CELL LAYERED MEASUREMENT

Predefined samples exist from different countries, languages, or user cells. At this level:

• cell results,
• weighted result,
• scope,
• uncertainty

reportable. Detailed rules for population weights and subgroup fairness are defined in later chapters of the book.

15. SYNTHETIC APPLE.COM DEMONSTRATION

SYNTHETIC METHODOLOGY DEMONSTRATION / The users, AI products, responses, rates, and results below are fictional. They do not represent the actual performance of Apple Inc. or the behaviour of any real AI product.

15.1. Scenario A — Single Positive Response

A synthetic user submits the following question to synthetic AI Product A: ‘What kind of company is Apple.com?’ Synthetic response: ‘Apple is a US-based technology company that develops consumer electronics, software and digital services.’ Preliminary evaluation:

• Entity identity is correct
• Main category is correct
• No obvious critical errors
• Basic representation is acceptable

Raw result:

1/1=100%

Incorrect public statement: “AI Product A represents Apple with 100% accuracy.” Correct public statement: “In the synthetic sample, the specified single observation met the transition criterion. Population prevalence and product reliability have not been determined.”

15.2. Scenario B — Discovery with Ten Observations

Ten independent synthetic users use the same version of the semantic prompt. Synthetic result:

• 7 responses pass
• 1 response is partially correct
• 1 response contains outdated information
• 1 response assigns the wrong main activity

raw pass rate:

7/10=70%

Under a simple, idealised assumption of independent sampling, the approximate 95 per cent Wilson interval is necessarily wide. The result remains exploratory because it is based on:

• small sample,
• single cell,
• single wave

Correct status:

EXPLORATORY — Not sufficient for the publicly available global NOMOS score.

15.3. Scenario C — 1,000 Observed Synthetic Wave

Assume synthetically that the following conditions are met:

• AI Product A
• 1,000 valid observations
• Predefined target population
• Locked prompt version
• Defined country and language layers
• Unmodified initial suitable response
• Full evidence records
• Predefined adjudication rules

Synthetic result:

Simple raw pass rate:

918/1000=91.8%

Approximate 95% Wilson interval under the idealised unweighted assumption: is. However, this number alone is not yet the final NOMOS GEO score. Because there are additional factors that need to be evaluated:

• Country and language weights
• Design effect
• Lowest language result
• Invalid observation rate
• Inter-rater agreement
• Verification of critical error candidates
• Untested user groups
• Time and user surface limitation of the measurement

Six Critical-error candidates must also undergo separate incident review. Correct reporting: ‘The raw response pass rate in the synthetic wave is 91.8 per cent. This result describes the observations before population weights and design effects are applied. Because six Critical-error candidates were identified, the conformity decision requires separate review.’ Incorrect reporting: ‘Apple's NOMOS GEO score is 91.8 and it is compliant.’ Synthetic data cannot be attributed in this way to a real company or a real system.

16. MANDATORY NORMATIVE PROVISIONS

N-01

A single generative system output cannot be used as the general GEO score.

N-02

A singular output observed alone:

• AI product and user surface,
• country,
• language,
• prompt,
• session,
• time

must be reported along with its conditions.

N-03

Single output:

• typical,
• continuous,
• global,
• belonging to all users

cannot be presented as behaviour.

N-04

Repetitions from the same user, account, or session cannot be counted as independent population observations without disclosing the dependency.

N-05

In the main population panel, the participant must record the first suitable output predefined; reproduction cannot be made to choose a positive response.

N-06

System rejection, error message, or inability to respond should be recorded as a valid system outcome if the protocol has been correctly applied.

N-07

Trimmed, modified, manually rewritten, or responses whose prompt context cannot be verified cannot be included in the main score calculation.

N-08

Selecting only positive screenshots and withholding the distribution of results is prohibited.

N-09

Synthetic observations cannot be used in real user or real system scores.

N-10

A population proportion should not be published without coverage, denominator, and appropriate uncertainty explanation.

N-11

The probability of a critical error cannot be quietly compensated within the overall average.

N-12

The observation result can only be applied to the inference universe that is uniquely defined; it cannot be generalised to unmeasured countries, languages, products, or time periods.

N-13

Measurement, scoring, or public statement must have an accountable human or institutional owner.

N-14

UNKNOWN, NOT TESTED, or INVALID records cannot be quietly converted into a positive result.

17. CANDIDATE GEO-1000 MAIN RULE

CANDIDATE-001

For every AI product being audited and for every main population measurement wave, a minimum of 1,000 valid user observations should be targeted by default.

This candidate sentence:

• accuracy of the general product forecast,
• cost,
• applicability,
• country and language fairness

This creates an initial balance among these considerations. It does not, however, guarantee adequate coverage of:

• all the country scores,
• all language scores,
• all user subgroups

Adequacy must instead be tested through:

• pilots,
• independent repetitions,
• design effect,
• subgroup sensitivity

Those evaluations determine the final normative status.

18. FORMS OF FAILURE

F-01 — Selected Screenshot

Only the most positive response is published among numerous results.

F-02 — Treating a Single User as the Population

The experience of one person is generalised to the entire target universe.

F-03 — Counting Repeats of the Same User as New People

Repeats generated by the same person or account are reported as independent participants.

F-04 — Renewing the Response and Picking the Best One

Initial or negative responses are discarded; the most appropriate response is saved as the measurement result.

F-05 — Prompt Drift

Participants change the word, tone, or scope of the prompt; results are combined as if from the same experiment.

F-06 — Context Contamination

Previous conversations affect the response; the record is presented as a clean session.

F-07 — Hiding Personalisation

Memory, special instructions, or user history is visible; the result is reported as a general user response.

F-08 — Confusing Model and Product

Only the general model or provider name is written; the actual product, interface, and tool condition are not recorded.

F-09 — Removing Errors and Mistakes from the Denominator

Deleted to make the results appear positive when the system does not respond to records.

F-10 — Counting Invalid Capture as System Failure

Reported as participant error, missing evidence, or incorrect outcome of the AI product.

F-11 — Globalising a Single Language and Country

The result of a single locale is applied globally.

F-12 — Using Synthetic Data as Real

Records created for demonstration purposes are added to actual performance.

F-13 — Losing a Critical Event on Average

Severe health, legal, licensing, or security errors are made invisible within a high average.

F-14 — Drawing Intervention Effect from a Single Success

The single positive response received after the GEO intervention is considered the causal effect of the study.

F-15 — Judging the Entire System from a Single Mistake

One error is generalised as if the AI product is always wrong and wrong for all users.

F-16 — Defining the Target Population Afterwards

After the outcome is seen, the user group with high success is declared the “actual target population.”

19. AUDIT PROCEDURE

An auditor should follow the sequence below when evaluating the validity of a single observation or an early GEO claim.

Step 1 — Record the Public Claim

The full proposed sentence is determined. Example: “ChatGPT correctly knows our company.”

Step 2 — Determine the Level of the Claim

The sentence may make a claim about:

• single event,
• prevalence,
• stability,
• global consequence,
• causality,
• commercial impact

The applicable level must be identified.

Step 3 — Determine the Supporting Observation Count

How many real users? How many accounts? How many independent sessions? How many repeats? How many AI products? How many languages and countries?

Step 4 — Verify Observation Conditions

Are the AI product, user surface, prompt, session, country, language and time all recorded?

Step 5 — Examine Evidence Integrity

Is there a full response? Is there a full screenshot? Has the response been selected or refreshed? Has the file been modified? Is the record synthetic?

Step 6 — Determine Dependency Structure

Are there repetitions of the same user? Has a shared account been used? Is there a mirrored model design? Is there clustering from the same panel?

Step 7 — Separate Data Collection Error from Valid System Result

Is the rejection or system error the actual result? Or did the participant violate the protocol?

Step 8 — Compare the Claim with the Level of Evidence

Is a single observation used for a population or global inference?

Step 9 — Open the Critical Path

If a single response carries a foreseeable severe harm, a separate incident investigation is initiated.

Step 10 — Limit the Decision

The public sentence is reduced only to the level carried by the evidence.

20. NECESSARY EVIDENCE

For an observation or claim within the scope of this section, the following records should be sought to the extent that they are relevant:

• Audited entity identity
• Fully disclosed GEO claim to the public
• Number of observations
• Number of unique participants
• AI product and user surface
• Visible model or version, if available
• Plan or account type
• Country and locale
• Prompt ID, version, and full text
• Session and personalisation status
• UTC timestamp
• Measurement wave
• Full raw response
• Screenshot or equivalent evidence
• Reproduction or answer selection status
• Observation validity status
• Dependency and matching record
• Synthetic or real status
• Invalid observation log
• Critical event record
• Result scope
• Uncertainty statement
• Responsible person or institution

Not all of these records may be required in the same detail for every low-risk discovery observation. However, if a broad GEO score or conformity clause is to be given to the public, material fields cannot be left blank.

21. AUDIT CHECKLIST

Has the audited entity been fully identified? Have the AI product and user surface been recorded? Was the prompt locked before measurement? Was the prompt changed by the participant? Are the session and personalisation conditions known? Is the user's country and locale information recorded? Is the full response preserved? Is it the first appropriate output or the best chosen output? Is there a complete record of evidence? Have the actual number of users and the number of observations been separated? Have repeated and matched observations been counted independently? Have refusals and system errors been properly managed in the denominator? Have invalid records been retained and justified? Have synthetic and real data been separated? Is a single response presented as a population result?

Is the claim applied only to the measured product, country, language, and time? Has the probability of a critical error been assessed separately? Does the result carry a statement of confidence or uncertainty? Are untested areas clearly shown? Does the public statement exceed the level of actual observation? If one of the factual questions is unanswered, the result carries limited confidence. If more than one is unanswered, it should not be used for the overall GEO score.

22. OBJECTIONS AND RESPONSES

Objection 1 — “The real user already sees only one answer.”

That is correct. The single answer seen by the single user is that person’s real experience. Therefore, individual events are recorded. However, individual experience is not the same as population experience. NOMOS maintains both facts together:

The experience of a single user is real. / A single user is not the entire population.

Objection 2 — “If the system gives the same answer every time, isn't one answer enough?”

The claim that the system gives the same answer every time also requires measurement. Additionally:

• time,
• retrieval,
• country,
• language,
• plan,
• personalisation,
• user surface

It may vary. Even a system assumed to be deterministic cannot be generalised beyond defined conditions.

Objection 3 — "Why exactly 1,000?"

The number 1,000 is not a magic threshold of certainty. GEO-1000 serves as a founding reference point that balances:

• a substantial user volume,
• a broad product-level estimate,
• practicable cost,
• public comprehensibility.

Subgroups may require more observations. Pilot studies must verify or revise the threshold.

Objection 4 — “Let the same 1,000 people test all AI products.”

This method can be useful for paired comparison across models. However, selecting only intensive AI users who can access all products may bias the target population. Also, observations from the same person are dependent. Fully paired, partially paired, and balanced designs will be compared later.

Objection 5 — “Isn’t a screenshot already sufficient proof?”

Screenshots are important. However, on their own they do not always show the following:

• real time
• country
• plan
• previous conversation
• personalisation
• answer refresh
• file change
• full response

For this reason, a screenshot is an important part of the evidence package. It is not the whole.

Objection 6 — “Why do we include a non-responding system in the score?”

The user's real experience may be not getting an answer. A rejection or error:

• technical,
• security,
• policy,
• access

may be the result. If the protocol was correctly applied, the event should be recorded. The system’s unresponsiveness due to the participant’s technical error is classified separately.

Objection 7 — “Why should a single severe mistake affect the entire inspection?”

A single mistake does not prove that most of the population made the same mistake. However, the potential damage of a single mistake can be severe. Therefore:

• score frequency,
• Critical decision harm and integrity

are evaluated separately. The average cannot automatically erase human harm.

Objection 8 — “If we can’t say anything with a single response, what is the value of early tests?”

Singular and small-sample tests:

• generate hypotheses,
• discover types of errors,
• test prompts,
• test the capture system,
• calibrate adjudicators,

can reveal critical events. What is wrong is not to do exploratory testing. What is wrong is presenting the exploratory result as a population score.

COMMON PRINCIPLE OF CHAPTER 24

This chapter was written to eliminate a single misconception: "I asked an AI, got an answer; here is my GEO result." No. You do not yet have a general GEO result. What you have is an observation. This observation:

• is real,
• is significant,
• even critical

may be. However, it is still a single observation. For a GEO score, the existence of:

• users,
• AI products,
• countries,
• languages,
• sessions,
• times

What must be measured is the distribution of representation across AI products, countries, languages, sessions and time. A single screenshot is not a distribution. Ten automated repetitions are not ten independent people. Nor do a thousand responses from a poor sample represent the world's population. Even a population-proportional sample may be incomplete without language fairness. A high average cannot erase Critical errors. NOMOS's first law of measurement is therefore:

A single answer is an event. / Multiple answers are a pattern. / A defined sample is an estimate of the population. / Repetition over time is a measure of stability. / But none of these alone are causality or sustainable value.

Order of Section 1 of NOMOS

Do not bring me one screenshot and then speak for the entire system.

Do not make the answer of one user the answer of the world's population.

Do not convert duplicates of the same account into independent humans.

Do not refresh the answer and display the preferred as if it were the real distribution.

Do not remove rejections, errors, and unknowns from the denominator.

Do not generalise one language to the world, one country to all countries, or one AI product to all generative systems.

Do not turn a single success into evidence of an intervention effect, or a single error into the immutable character of the entire system.

Do not drown a single severe error within the average.

First, record the event. / Then measure the pattern. / Then define the target population. / Then show dependency and uncertainty. / And speak only to the extent that the evidence carries.

The Chapter's Closing Sentence

The representation of an entity in artificial intelligence systems is not a single answer. AI products are a distribution that should be measured across users, countries, languages, sessions, and times.

Normative Core

A single generative-system output MAY establish that a specific response occurred under recorded conditions. A single output MUST NOT be used to infer: - population prevalence, - typical product behaviour, - temporal stability, - cross-language consistency, - cross-country consistency, - cross-product consistency, - intervention effect, - recommendation certainty, - commercial impact, - conformity, - or a GEO score. All broader claims MUST be supported by a predefined target population, valid multi-observation design, recorded dependence structure, evidence bundle, evaluation rule, scope statement, and uncertainty disclosure. A single high-harm output MAY trigger a separate critical-incident review without being treated as evidence of population prevalence.