Representation Is a Distribution

Chapter Boundary

Section 1 established the following provision:

A single response is not the GEO score. A single response is a singular observation that occurred under recorded conditions.

This section answers the next question:

If a single response is not the GEO score, what is the actual object we want to measure?

The answer is:

The representation of an entity in generative systems is a multidimensional distribution formed across AI products, users, countries, languages, session conditions, prompts, and times.

This chapter:

• the representation distribution,
• the fundamental dimensions of the distribution,
• the difference between the mean and the distribution,
• the subgroup inequality,
• the stability,
• the coverage and missing observation problem,
• the risk of rare severe errors,
• the conceptual structure of the publicly available result card

defines. This chapter does not yet:

• how many participants will be assigned to each country,
• how the sample frame will be established,
• how prompt translations will be validated,
• how atomic claims will be scored,
• the exact formula of the final NOMOS score,
• conformity thresholds

This chapter does not settle these matters normatively; later chapters address them separately. Before defining the scoring formula, Chapter 2 defines the reality that the score is intended to represent.

NOMOS Challenge

Consider two AI products. Each receives the same global average score:

90 / 100

The first product's language outcomes are as follows:

• English: 90
• Turkish: 91
• German: 89
• Japanese: 90
• Arabic: 90

The language results of the second product are as follows:

• English: 99
• Turkish: 98
• German: 97
• Japanese: 86
• Arabic: 70

The simple average of both products is 90. Do they have the same GEO quality? No. The first product produces a similar representation in different languages. The second product produces excellent representation in some languages but seriously weak representation in another language. A single average hides this difference. Now consider two other products. 95% of the responses of both meet the passing criteria. The remaining 5% in the first product consists of:

• small scope deficiencies,
• limited explanation shortcomings,
• a few UNKNOWN

results. The remaining 5% in the second product consists of:

• incorrect health authority,
• nonexistent licence,
• incorrect legal identity,
• inappropriate advice

exists. The raw accuracy rate for both is again 95 per cent. Do they have the same representation risk? No. Now consider a third pair. The average of both across the three measurement waves is 90. The first one:

• Wave 1: 90
• Wave 2: 91
• Wave 3: 89

The second one:

• Wave 1: 98
• Wave 2: 90
• Wave 3: 82

Do they have the same consistency over time? No. In that case, the issue is not only: 'What is the average?' We also need to know the following: How are the results distributed? Who is well represented, who is poorly represented? Which languages and countries are falling behind? How severe are the errors? How much does the result change over time? Which areas were never measured? With what weights was the average created? How many different realities lie behind the same average? The first conclusion of this section is:

The same average is not the same representation.

Its second provision states:

a GEO score may be a summary of the distribution; it cannot replace the distribution.

1. PURPOSE OF THE CHAPTER

This section defines how the representation of entities in generative systems should be understood before being reduced to a single number. Its main objectives are:

• To show that representation is not a timeless and singular “AI perspective"
• To define the conditions on which the representation outcome depends
• To explain that the true–false binary alone is insufficient
• To separate average accuracy from subgroup inequality
• To make visible the heavy error risk hidden within high averages
• To show that stability and accuracy are not the same concept
• To prevent unmeasured areas from being filled with positive assumptions within the distribution
• To prepare the structure of a future NOMOS score that is singular but not condemned to being singular

At the end of this section, the reader should be able to answer the following question:

Why is the result of an entity's GEO not limited to only the average accuracy rate?

2. CENTRAL NORMATIVE PROVISION

The representation of an entity in generative systems is the conditional distribution of different responses and representation outcomes that can occur under defined conditions.

This distribution:

• it is not the abstract 'thought' of a single model,
• it is not the common view of all artificial intelligence systems,
• it is not only the repetition rate of the website,
• it's not only the frequency with which the brand's name is mentioned,
• it's not just the average accuracy,

it is not only the recommendation rate. The representation distribution, within a certain protocol, shows the following:

Which users see what kind of representation on which AI product and user surface, in which language, in which country, with which prompt, under which session condition, and at what time.

Therefore, the result of a distribution is always dependent on the following questions: Which entity? Which AI product? Which user surface? Which target population? Which country? Which language and locale? Which prompt? Which session condition? Which measurement wave? Which Truth Pack? Which evaluation rule? When any of these fields change, the distribution may also change.

3. WHAT DOES THE WORD "DISTRIBUTION" MEAN?

In this book, the word distribution does not refer to a hidden or metaphysical view that is assumed to exist within an artificial intelligence system. It does not claim: NOMOS: "This is the real distribution of the company in the model's mind." The distribution here is operational and observable.

It is the pattern of different representation results that can be obtained under a defined protocol within the target population and measurement conditions.

Two distributions should be distinguished.

3.1. Target Representation Distribution

It is the distribution that the measurement tries to predict but cannot directly observe in full. It is represented as:

P_E(Y | A, C, L, S, P, T, U)

Here:

• E: audited entity
• Y: representation result
• A: AI product and user surface
• C: country or geographic condition
• L: language and locale
• S: session and personalisation condition
• P: prompt
• T: time or measurement wave
• U: target user universe

This expression asks: “In the defined target user universe, under the specified conditions, which representation results occur with what probabilities?” Not all elements of the target distribution can be measured. They are estimated through sampling.

3.2. Observed Empirical Distribution

It is the distribution of valid observations obtained in real or synthetic measurement. Its representation:

P̂_E

can be used. The empirical distribution is not the target distribution itself. It is an estimate produced through its sample, weighting, and measurement design. The magnitude of the difference may depend on:

• Sample design
• Participant selection
• Country and language coverage
• Invalid observation rate
• Non-response
• Adjudicator error
• Measurement wave
• Prompt selection
• Product and interface conditions
• Weighting
• Clustering and dependence

Therefore, an empirical distribution cannot be published as "the absolute and immutable truth of artificial intelligence." The correct expression should be: "The estimate obtained for the specified target population and measurement conditions."

4. REPRESENTATION SPACE

The representation of an entity does not occur in a single dimension. Each observation exists in a multidimensional condition space. The basic representation:

X_i = (E_i, A_i, U_i, C_i, L_i, S_i, P_i, T_i)

Here:

• Ei: entity
• Ai: AI product and user surface
• Ui: user or sampling unit
• Ci: country and geographical layer
• Li: language and locale
• Si: session, plan, and personalisation condition
• Pi: prompt or prompt family
• Ti: time and measurement wave

Generated response:

R_i

can be shown. The representative state resulting from the evaluation of the response is:

Y_i = g(R_i, V_E, J)

and can be considered as. Here:

• VE: verified Truth Pack used for the entity
• J: assessment and adjudication rules
• g: evaluation process that converts the response into representative results

The meaning of this representation is as follows: The same response cannot be scored on its own without indicating which entity, which Truth Pack, and which evaluation rule it was examined according to.

5. REPRESENTATIVE RESULT IS NOT A SINGLE VALUE

Only the entirety of a response:

• can be forced into the categories of true,
• is wrong

Giving a label is often insufficient. The following situations can occur together in the same response:

• Correct identity
• Correct main category
• Incomplete service coverage
• Outdated number of employees
• Unsupported leadership claim
• Correct citation
• Inappropriate recommendation
• Non-critical boundary deficiency

Therefore, the representation result can be considered as a result vector. Candidate representation:

Y_i = (y_i^id, y_i^cat, y_i^fact, y_i^scope, y_i^time, y_i^evidence, y_i^recommendation, y_i^harm)

Here:

• yiid: identity accuracy
• yicat: accuracy of category and role
• yifact: accuracy of factual matters
• yiscope: preservation of scope and boundaries
• yitime: temporal validity
• yievidence: claim–evidence consistency
• yirecommendation: recommendation and user suitability
• yiharm: harm class of error or misguidance

The exact scoring rules for these fields will be defined in later sections. The guiding principle here is:

A representative answer should be evaluated not only on whether it contains correct information but also on whom, what, within which boundaries, with what evidence, and for whom it is appropriately conveyed.

6. BASIC REPRESENTATION STATES

The status of each atomic claim and final response may not be the same. This section conceptually separates the following basic states.

6.1. SUPPORTED

The claim is supported by sufficient evidence within the defined scope and time.

6.2. CONTRADICTED

The claim conflicts with reliable and relevant records.

6.3. UNSUPPORTED

There is not enough support for the claim. It has not been proven that the claim is necessarily false. However, there are not enough records for it to be accepted as true.

6.4. OUTDATED — NO LONGER CURRENT

The claim may have been true in the past. As of the measurement date, it is no longer valid or its current accuracy cannot be verified.

6.5. PARTIALLY SUPPORTED — PARTIALLY SUPPORTED

Only a specific scope, population, country, product, or time of the claim is supported. The response may have exceeded the support limit.

6.6. OUT OF SCOPE — OUT OF SCOPE

The claim is not related to the audited entity or Truth Pack. It may have come from another entity, product, country, or service.

6.7. ENTITY CONFLATION — ENTITY MERGER

System:

• two different companies,
• the brand and the legal entity,
• a person and the institution,
• old and new entity,
• has confused different organisations with the same name

with each other. This error is not merely a factual mistake. It is the corruption of the representation object.

6.8. MATERIAL OMISSION — MISSING INFORMATION

The sentences in the response may be correct. However, essential information that would change the user's decision has been omitted. Example: The service is offered only in certain countries. The licence is limited to a specific jurisdiction. Price is excluding tax. The result is based on first-party data. The product is not suitable for certain users. The omission can create a wrong inference even if there is no directly incorrect sentence.

6.9. REFUSAL — REJECTION

The system has refused to answer. Rejection:

• appropriate,
• unnecessary,
• inconsistent,
• security-based,
• technical

may occur. Ret is not automatically considered failed or successful. It is classified separately.

6.10. NO RESULT

The system did not produce a usable response due to technical or operational reasons. It is not the same as Ret.

6.11. UNKNOWN

The available records are not sufficient for a reliable classification. UNKNOWN cannot be converted into a positive result.

7. MAIN CHARACTERISTICS OF THE DISTRIBUTION

A representation distribution cannot be explained solely by the mean. At a minimum, the following characteristics should be evaluated.

7.1. Central Tendency

Summarises the general accuracy or transition level of the responses. Example measures:

• Mean
• Weighted mean
• Median
• pass rate

Central tendency is important. However, it only shows the centre of the distribution.

7.2. Spread and Variability

Indicates how much the results are spread around the centre. Of two systems with the same mean, one may be consistent among users, while the other may be highly variable. Variability:

• users,
• countries,
• languages,
• sessions,
• measurement waves

can also be examined among them.

7.3. Shape of the Distribution

Are the results clustered around a single central point? Or are they divided into two or more clusters? For example:

• If half of the users see very accurate,
• and half see very inaccurate

responses, the average may be medium. However, the typical user experience is not "medium accuracy." It is two separate polarised experiences.

7.4. Subgroup Differences

Do specific language, country, or user groups systematically see different results? The global average may hide this difference.

7.5. Tail Risk

Are there low-frequency but severe errors? Rare events can have a small effect on the average. Their impact on human harm can be significant.

7.6. Coverage

How much of the target user universe has actually been measured? A high score can be found together with low coverage.

7.7. Missing and Unknown Mass

What proportion of the responses are:

UNKNOWN,

NOT TESTED,

rejection, technical non-response or an invalid record?

7.8. Time Stability

To what extent is the distribution preserved across different measurement waves?

8. CANDIDATE NOMOS REPRESENTATIVE DISTRIBUTION SIGNATURE

CANDIDATE CONCEPT

It is recommended that the result of a GEO be defined first by the following distribution signature instead of a single number:

RDS(E) = (μ, κ, σ, Δ, τ, ρ, u)

This representation is called Representation Distribution Signature, that is:

Representation Distribution Signature

Components:

μ — General Representation Level

Overall accuracy or transition prediction under defined weights and scope.

κ — Coverage

The extent to which the target user, country, language, AI product, and time universe are sufficiently measured.

σ — Variability

Spread among users and experimental cells.

Δ — Subgroup Inequality

The material difference between language, country, product, or other significant groups.

'tau — Tail and Critical Error Risk

The rate and nature of rare severe errors.

ρ — Temporal Stability

The resilience of distribution across waves.

u — Unknown and Unresolved Mass

UNKNOWN is the share of the distribution that remains unresolved or insufficiently assessed. This signature is not yet a final scoring formula. The following elements require further development:

• definite definition,
• estimation,
• weight,
• public presentation

They will be finalised through pilot studies and later chapters. The founding principle is:

The GEO result should answer not only the question 'how accurate?' but also 'for whom, how consistent, how equitable, how comprehensive, and how risky?'

9. AVERAGE WASHING

If an institution only shows a high global average and hides the material flaws of the distribution, this is called:

Average washing

Average washing can occur in the following ways:

• Hiding the weakest language
• Excluding small countries from the report
• Mixing critical errors into the general error rate
• Removing UNKNOWN results from the denominator
• Only including well-performing AI products
• Excluding the weak measurement wave
• Making the population outside the scope invisible
• Changing weights after seeing the results

The average may be mathematically calculated correctly. However, the representation it offers to the public can still be misleading. Therefore:

A mathematically correct calculation of an average does not mean it is sufficient for decision-making.

10. DIFFERENT REALITIES OF THE SAME AVERAGE

The average could be 90 in all of the following synthetic distributions.

10.1. Homogeneous Distribution

All major subgroups are in the range of 88–92. This distribution is relatively balanced.

10.2. Polarised Distribution

Half of the users see results close to 100, while the other half see results close to 80. The average is 90. However, the user experience is not equal.

10.3. Linguistically Fragmented Distribution

Results are very good for high-population or high-resource languages, but significantly weak for low-resource languages. The global average may look good. Language fairness is weak.

10.4. Country-Based Fragmented Distribution

Even within the same language, results may vary due to different countries, locales, product access, or user base.

10.5. Temporally Volatile Distribution

Very high results occur in one wave, very low results in another wave. Even if the average is high, it is difficult to establish reliable user expectations.

10.6. Distribution with Tail Risk

Most of the responses are correct. However, a small portion contains serious health, legal, licensing, or safety errors.

10.7. High Unknown Mass Distribution

Most of the assessable responses are correct. However, a significant portion of the target population or responses is:

• tested,
• untested,

UNKNOWN

or inconclusive. Publishing a high rate based only on the assessable responses may obscure the scope.

11. SUBGROUP INEQUALITY

A global score may not represent the experience in subgroups. Key subgroups may include:

• AI product and user surface
• Country
• Language
• Locale
• Plan
• Web or mobile
• Controlled or natural panel
• Time wave
• User intent
• Device family

Not all of these groups may have the same importance in every audit. Material groups should be determined when scope is locked.

11.1. Language is Not the Same as Country

Turkish-speaking:

• User in Turkey,
• User in Germany,
• User in another country

They can use the same language. However:

• product access,
• price,
• resources,
• local interface,
• legal context,
• personalisation

may be different. Therefore, all differences cannot be explained solely by language or country.

11.2. Intersectional Cells

Some material differences only become visible when multiple areas are examined together. Example: Germany × Turkish × Mobile × Free plan × Natural User Panel could in itself be an experimental cell. However, as cells multiply, the sample size decreases. Therefore, a definitive score cannot be published for each small cell. Measurement design: should preserve meaningful subgroups and should not produce statistical false precision.

11.3. Subgroup Base

CANDIDATE CONCEPT

In addition to the global average, it is recommended to report the result of the weakest adequately sampled subgroup or the bottom 10% group separately. This can be referred to as:

Representation base

or:

Language/Country Floor

However, if the "lowest score" comes from a small and unstable cell, it may produce incorrect results. Therefore, the base metric should be defined together with:

• minimum sample,
• confidence interval,
• cell quality

The exact rule will be determined in the following sections.

12. IT IS PART OF THE SCOPE DISTRIBUTION

A distribution consists not only of observed responses. What was never observed also determines the meaning of the result. For example, English and Turkish were tested; Japanese and Arabic were not. The web interface was tested; the mobile interface and paid plan were not. Strong English and Turkish results do not justify the statement, ‘Representation is strong across all languages and user surfaces.’ The defensible conclusion is: ‘Measurements were conducted on the specified English and Turkish web surfaces. Other languages and mobile surfaces were not tested.’

12.1. Coverage Map

Each GEO distribution should carry the following map to the extent relevant:

• Target universe
• Sampling frame
• Covered countries
• Non-covered countries
• Covered languages
• Non-covered languages
• Monitored AI products
• Unmonitored products
• Monitored user surfaces
• Unmonitored surfaces
• Measurement waves
• Out-of-scope prompt families

12.2. Coverage Rate

CANDIDATE CONCEPT

A coverage metric can be developed to show what proportion of the target universe is represented with sufficient measurement. However, coverage:

• only the number of countries,
• only the number of languages,
• only the population ratio

cannot be measured. For example, 150 countries may be covered. However, 95% of the participants may have come from only five countries. Or 90% of the population may be covered, but small languages may be entirely excluded. Therefore, population coverage should be shown separately from country/language observer coverage.

13. MISSING OBSERVATION AND NON-RESPONSE

Missing data is not only a technical issue. Its distribution can be systematically distorted. The reasons for an observation being missing may include:

• Participant's failure to complete the task
• Product not being accessible in the country
• Computation constraints
• Technical error
• System rejection
• Invalid screen capture
• Participant changing their prompt
• Product not working in a specific language
• Adjudicators being unable to make reliable decisions

Each of these reasons requires a separate status.

13.1. Missing Data May Not Be Random

For example, if more tasks cannot be completed in countries with low connection quality, using only successful recordings can shift the global result in favour of wealthy users with strong connections. If the rate of UNKNOWN rises because answers in a specific language are difficult to evaluate, the score calculated from only known answers may hide language inequality. Therefore:

The missing data rate and the distribution report of the reason for missing data should be part of the report.

13.2. Denominator — Integrity of the Denominator

A report should answer the question clearly: "90 per cent of exactly what is 90 per cent?" Possible denominators:

• Assigned participants
• Task initiators
• Task completers
• Protocol-compliant records
• Valid system results
• Responses with adjudicator decisions
• Specific language or country cell

The denominator should not be changed after the result is seen. For example:

• 1,000 tasks assigned,
• 900 completed,
• 800 valid,
• 720 passed

Thus:

720/800=90%

This is the pass rate among valid records. The observed pass rate across the entire assigned sample is instead:

720/1000=72%

answers a different question. Both rates can be used. They cannot be presented in the same sense.

14. ACCURACY AND CONSISTENCY ARE NOT THE SAME

A distribution may have high accuracy. However, it may be inconsistent over time or across users. Another distribution may have low accuracy. However, it may consistently be wrong. Therefore, four situations should be distinguished.

Being “consistent” is not automatically positive. The system may be quite consistent if it gives the same error to every user.

14.1. Inter-User Consistency

Do different users see similar material representation in the same cell?

14.2. Inter-Session Consistency

Does the same or matched user see similar results in new sessions?

14.3. Inter-Wave Consistency

Does the same protocol produce a similar distribution on different days or periods?

14.4. Post-Product Update Consistency

When there is a material change in the AI product, does the previous distribution continue?

15. PRODUCING THE SAME SENTENCE IS NOT STABILITY

Stability: It does not mean "giving the exact same answer to every user word for word." An AI product can use different expressions, order, and examples. As long as they preserve the material meaning, they can be acceptable variations. The important distinction is:

Formal Variability

Word choice Sentence order Length Tone Example Using list or paragraph If material truth is preserved, it may not be an error.

Semantic Variability

Change of company identity Differentiation of the main category Expansion or contraction of the field of activity Change of licence or competency Change of price and geography Recommending to one user but not recommending to a similar user without explanation If the material representation changes, there may be a stability problem. Therefore, the preferred concept is:

Conditional semantic stability

It should be. The fundamental reality is expected to be preserved under equivalent material conditions. A change in recommendation due to different user needs can be appropriate contextual behaviour.

16. CONTEXT SENSITIVITY AND ARBITRARY VARIABILITY

It is not always correct for a system to give the same recommendation to all users. For example:

• Users with a low budget
• Only users seeking services in a specific country
• Users requiring a different licence
• Users seeking individual rather than corporate services

can receive different conformity outcomes for the same entity. This difference:

• may be due to the user condition,
• the scope of service,
• or the actual limit

and may be context-sensitive and accurate. However, if the outcome changes without reason under equivalent user conditions, it may be arbitrary variability. Therefore, NOMOS does not want: “The same answer for all users.” It wants:

A response that is consistent and explainable under equivalent material conditions and sensitive to actual limits under different conditions.

17. TIME IS WITHIN THE REPRESENTATION DISTRIBUTION

The representation distribution is not timeless. It can be thought of as PE,t. The distribution of the same entity:

• product update,
• resource change,
• company change,
• news event,
• new language version,
• technical access change

can diverge afterwards.

17.1. Sudden Change

After a product or resource update, the distribution changes quickly in a short time.

17.2. Gradual Change

Due to new resources and user behaviours, the result slowly changes over time.

17.3. Temporary Fluctuation

During a specific event or news period, the distribution temporarily differs.

17.4. Post-Intervention Change

After the GEO intervention, the distribution may change. However, claiming that the change is due to the intervention also requires a baseline and causality design.

17.5. Smoothing with Time Average

For a product:

• in the first wave 98,
• in the second wave 90,
• in the third wave 82

If taken, the average is 90. However, publishing 90 hides the downward trend. Therefore:

The time average cannot replace the wave results.

18. TAIL RISK AND HEAVY ERROR DISTRIBUTION

A large portion of a distribution may be correct. However, low-probability heavy errors should also be evaluated. Examples:

• Health licence that does not actually exist
• Incorrect emergency guidance
• Incorrect legal authority
• Fake investment guarantee
• Incorrect company identity
• Inappropriate advice for children or vulnerable groups
• Repetition of the manipulative instruction hidden from people

In the previous architecture:

• Critical,
• Major,
• Moderate,
• Advisory

were separated into four levels of importance, and it was suggested that Critical findings should have an automatic failure effect. This book links this system to population distribution.

18.1. Separation of Frequency and Severity

An error:

• frequent but low impact,
• rare but severe

may occur. However, the frequency rate alone cannot evaluate the two cases equally.

18.2. Candidate Severity-Weighted Risk Representation

CANDIDATE CONCEPT

A risk indicator can be developed using severity coefficients for error classes:

tau = Σ_{k=1}^K λ_k p_k

Here:

• pk: predicted rate of the error type
• λk: severity coefficient of the error type

However, coefficients cannot be determined arbitrarily. They should be based on:

• Foreseeable human harm
• Representation integrity
• Impact on user decision
• Reversibility
• Intent and manipulation
• High-risk area

The precise method will be defined in Section 15.

18.3. Critical Gate

A high overall average cannot automatically delete a verified Critical error candidate. The correct public record should show both results together: “The overall representation estimate is high.” and: “The specified Critical event or events also affect the conformity decision.”

19. REFUSAL AND FAILURE IN DISTRIBUTION

A system not responding is also part of the user experience. However, not all refusals are the same.

19.1. Appropriate Refusal

The system may exercise caution when evidence is insufficient or when it cannot provide a safe response. This can be the correct behaviour in some contexts.

19.2. Unnecessary Refusal

It may inconsistently avoid responding to publicly available and low-risk information.

19.3. Technical Nonresponse

The system may not produce a response due to a technical error.

19.4. Out-of-Scope Rejection

The prompt may be incorrect or outside the task protocol.

19.5. Preservation of Rejections in Distributions

Rejections:

• success,
• failure,

UNKNOWN

should not be forced into a single category. Their function and justification should be evaluated. Removing rejections from the denominator may yield a high accuracy rate only among successful responses and obscure the real user experience.

20. HOW CAN DISTRIBUTIONS BE COMPARED?

Simply comparing the result numbers is not sufficient to compare the distribution of two AI products. For the comparison to be fair, the following areas should be as equal as possible or explicitly modelled:

• Audited entity
• Target population
• Prompt version
• Languages
• Countries
• Measurement wave
• Session condition
• User surface
• Plan distribution
• Adjudication rule
• Reality package
• Validity rules
• Weights

Two scores obtained with materially different protocols cannot be used directly for ranking.

20.1. Common Support Comparison

All compared AI products:

• are officially accessible,
• support the same language and user conditions

can be compared in a common universe. This provides a cleaner comparison between systems.

20.2. Natural Access Distribution

Each AI product can be measured in its own real user access universe. This result shows the product's actual global impact. However, since target populations may differ, direct performance ranking requires caution.

20.3. Different Surfaces of the Same Provider

Web, mobile, enterprise products, or different plans, even if they belong to the same provider, can create separate distributions. If they are to be combined under a single result, the combination and weighting method should be explained.

21. WEIGHTING CHANGES THE MEANING OF THE DISTRIBUTION

A global average depends on the weight with which observations are combined. Possible weights:

• Equal by country
• By country population
• Appropriate AI user population
• Language user
• Actual product exposure
• Equal AI product
• Market share
• Customer target market

Each method answers a different question.

21.1. Equal Country Weight

San Marino and India can receive the same weight. This approach can address the question of 'Average country experience.' It does not represent the average user experience in the world population.

21.2. Population Weight

Larger countries have more influence. This approach can approach the global population experience. It may render small countries and languages invisible.

21.3. Language Fairness Weight

It can also ensure that low-resource languages remain visible. It is not the same as the global population score.

21.4. Product Exposure Weight

The weighting of AI products according to actual user exposure can be considered. It requires reliable and independent usage data.

21.5. Principle of Normative Weighting

Weights should be defined before the results are seen, publicly disclosed, and the question they answer should be specified.

It is prohibited to choose a weighting system that will increase the positive score after the result is seen. Alongside a single global score, alternative meaningful weighted results can also be published.

22. SCORE DISTRIBUTION IS LOSSY COMPRESSION

It is possible to produce a single number from a distribution. However, a single number does not carry all the information in the distribution. Therefore, the score:

Is a lossy summary of the distribution

should be accepted. A score may lose the following information:

• Weakest language
• Weakest country
• Critical error type
• Unknown mass
• Time fluctuation
• Polarisation
• Product and plan difference
• Out-of-scope population
• Rejection distribution
• Adjudicator ambiguity

The score is useful because:

• facilitates comparison,
• simplifies public communication,
• allows monitoring change,

supports the management decision. However, the simplicity of the score cannot turn into the elimination of distribution.

23. MINIMUM PUBLIC RESULTS CARD

CANDIDATE REQUIREMENT — CANDIDATE OBLIGATION

If a global score NOMOS is to be presented to the public, it is recommended that at least the following results be shown alongside it:

The main score should not be used as a standalone advertising label. The canonical record must be linked to this results card.

24. SYNTHETIC APPLE.COM DISTRIBUTION DISPLAY

SYNTHETIC METHODOLOGY DEMONSTRATION / The AI products, users, languages, numbers, and results below are entirely fictional. They do not represent the actual performance of Apple Inc. or any real AI product. Let us consider two synthetic AI products:

• Product A
• Product B

For each product:

• five languages,
• 200 valid synthetic observations per language,
• a total of 1,000 observations

are assumed. This equal language distribution is only to demonstrate the method. It does not represent the actual GEO-1000 population weighting.

24.1. Language-Based pass rates

The average of the two products is the same. However, their distributions are different.

Product A

Language results are close to each other. The difference between the highest and lowest language is 2 points. The language floor is 89%.

Product B

Produces very high results in three languages. There is a decline in Japanese and especially in Arabic. The difference between the highest and lowest language is 29 points. The language floor is 70%. Note: Saying "Both scored 90" hides the material difference.

24.2. Response Status Distribution

The pass rate of the two products is again 90%. However: In Product B, the number of Critical error candidates is 14 times higher. In Product B, the unknown and inconclusive mass is higher. A larger portion of failures in Product A is due to partial support deficiency. The error backlog of Product B is heavier. The same pass rate does not necessarily result in the same fitness outcome.

24.3. Time Waves

Product A is relatively stable. Product B shows a noticeable decline. Publishing only the average of the three waves hides the deterioration in Product B.

24.4. Synthetic Result

It may be possible to publish the same headline score for Product A and Product B:

90

But NOMOS result cards cannot be the same.

Product A — Synthetic Distribution Signature

Overall result: high Language inequality: low Critical error tail: low but not zero Unknown mass: low Time stability: high

Product B — Synthetic Distribution Signature

Overall result: high Language inequality: high Critical error tail: substantial Unknown mass: higher Time stability: low and declining Therefore, the synthetic conclusion of Section 2 is as follows:

Having the same average score for Product A and Product B does not indicate that they have the same quality of representation.

25. MANDATORY NORMATIVE PROVISIONS

CH02-N01

A GEO result cannot be defined only by a single average or a single pass rate.

CH02-N02

Each distribution result:

• the audited entity,
• target population,
• AI product and user surface,
• country and language,
• The prompt,
• the residence condition,
• to time

should be connected.

CH02-N03

The global average alone cannot compensate for material subgroup inequality.

CH02-N04

Critical and other high-severity error outcomes should be reported separately from the overall error rate.

CH02-N05

UNKNOWN, no-result, rejected, invalid, and NOT TESTED statuses should be distinguished from each other.

CH02-N06

Unmeasured country, language, product, user surface, or time period cannot be shown as part of the positive distribution.

CH02-N07

Weights and aggregation rules should be defined before viewing results.

CH02-N08

Post-result weight changes should create a new analysis version, and previous results should be preserved.

CH02-N09

Distributions based on materially different target populations, demand, product, time, or evaluation criteria cannot be directly compared without explanation.

CH02-N10

Formal language variability cannot automatically be counted as representation error.

CH02-N11

Material identity, scope, time, competence, or recommendation variability cannot be classified as a style difference without justification.

CH02-N12

High stability alone is not evidence of quality; consistently wrong results should also be recorded.

CH02-N13

A high average, if accompanied by low coverage, should have public results with coverage limitation.

CH02-N14

Subgroup scores should not be published as definitive country or language judgements under insufficient sampling and broad uncertainty.

CH02-N15

Rejection and technical failure rates should be preserved as part of the genuine user experience.

CH02-N16

The only score made public should be linked to the canonical distribution and scope record.

CH02-N17

Synthetic distributions cannot be combined with the performance of a real AI product or real entity.

CH02-N18

The distribution report should have an accountable human or institutional owner.

26. FORMS OF FAILURE

CH02-F01 — AVERAGE THINKING

High global average is shown; language, country, or Critical error corruption is stored.

CH02-F02 — DELETE SUBGROUP

Low-performing groups are removed from the report or declared 'insignificant'.

CH02-F03 — SCOPE MAP

Narrow-scope good results are used for the whole world, all languages, or all products.

CH02-F04 — SEVERITY NORMALISATION

Minor deficiencies and serious health or licensing errors are combined within the same error count.

CH02-F05 — HIDE DETERIORATION WITH TIME AVERAGE

Decline or fluctuation is made invisible only with the period average.

CH02-F06 — WORSHIP OF STABILITY

A consistent answer is assumed to be automatically correct.

CH02-F07 — WORD-FOR-WORD SAME ANSWER FALLACY

Literal text matching is used as a measure of consistency instead of semantic correctness.

CH02-F08 — COUNTING CONTEXTUAL DIFFERENCE AS ERROR

Differences in appropriate recommendations based on actual user needs are marked as arbitrary variability.

CH02-F09 — COUNTING ARBITRARY VARIABILITY AS PERSONALISATION

Material contradictions under equivalent user conditions are defended as “personalisation” without explanation.

CH02-F10 — CHANGING THE DENOMINATOR

Post-result transition is made among completed, valid, or evaluated observation denominators to elevate the positive outcome.

CH02-F11 — POST-RESULT WEIGHTING

The country, language, or model coefficients producing the highest score are selected afterward.

CH02-F12 — MERGING INCONSISTENT CELLS

Different prompt, product, time, or panel conditions are presented as a single distribution.

CH02-F13 — DELETING REJECTIONS

System records that did not respond are removed to generate high accuracy only among those that produced a response.

CH02-F14 — DEFINITE JUDGEMENT FROM SMALL CELL

A country or language score is produced from one or several users.

CH02-F15 — HIDING THE CRITICAL TAIL

Rare severe errors are lost within the overall pass rate.

CH02-F16 — SINGLE SCORE REPORT

Only the headline score is published without showing scope, inequality, stability, uncertainty, and severe errors.

CH02-F17 — COUNTING DISTRIBUTION AS "AI'S BELIEF"

Observed user surface results are presented as if they are the invariant and intrinsic viewpoint of artificial intelligence.

CH02-F18 — COUNTING THE UNMEASURED AS ZERO VARIANCE

Untested areas are considered smooth and stable.

27. AUDIT PROCEDURE

Whether a GEO report honestly represents the distribution should be examined with the following steps.

Step 1 — Record the Public Claim

Example: “The brand's global NOMOS score is 92.” The full sentence and the surface where it is found are recorded.

Step 2 — Define Target Distribution

For which score:

• user universe,
• AI products,
• countries,
• languages,
• time period

is it applied?

Step 3 — Extract Measurement Cells

Each material combination is determined:

• Product
• Country
• Language
• Panel
• Prompt
• Wave
• User surface

Step 4 — Rebuild Denominators

Assigned tasks, completed tasks, valid records, adjudicated records and passed responses must be reported as separate counts.

Step 5 — Extract Result Statuses

Not only the pass rate:

• partial,
• unsupported,
• conflicting,
• old,
• Critical,

UNKNOWN,

rejected, inconclusive distribution is examined.

Step 6 — Verify Weights

Weights:

• are they predefined,
• which question do they answer,

have they changed after the result?

Step 7 — Examine Subgroup Differences

The best group, the weakest sufficient group, language difference, country difference, product difference, and panel difference are evaluated.

Step 8 — Review Scope and Deficiency

Which areas were not measured? Which cells are insufficient? What are the reasons for the deficiency? Could the missing data be systematic?

Step 9 — Examine the Determination

The change between Waves Sessions Users Interfaces is evaluated.

Step 10 — Examine Queue Risk

Critical and other serious errors:

• number,
• its rate,
• type,
• user effect

is evaluated separately in terms of

Step 11 — Compare the Public Summary with the Distribution

Does the single number or sentence published to the public accurately convey the direction and risk of the full distribution?

Step 12 — Limit the Result

The public claim is limited only to the level carried by the distribution and scope.

28. NECESSARY EVIDENCE

To the extent relevant for a representative distribution or a global GEO score, the following records should be searched:

• Audited entity record
• Definition of target population
• Sampling frame
• AI product and user interfaces
• Country and language cells
• Prompt and versions
• Session and panel conditions
• Measurement waves
• Assigned task numbers
• Completed tasks
• Valid and invalid records
• Raw responses
• Result status distribution
• Adjudication records
• Critical error records
• Rejection and inconclusive records
• UNKNOWN records
• Weights
• Stakeholders
• Subgroup results
• Stability results
• Out-of-scope areas
• Confidence or uncertainty calculations
• Public scorecard
• Responsible person or institution
• Protocol and analysis version
• Change history

29. AUDIT CHECKLIST

Is the distribution's target population clear? Are the audited AI products and user surfaces specified? Are the country, language, prompt and time scopes visible? Are the weights used to produce the composite score disclosed? Is the denominator fully defined? Are invalid records reported separately? Are refusals and inconclusive outcomes preserved? Are UNKNOWN and NOT TESTED distinct? Are subgroup results visible? Have small cells been interpreted with excessive certainty? Has the weakest material group been retained? Are Critical errors reported separately? Are measurement waves shown separately? Does the average conceal decline or volatility? Is a merely formal difference distinguished from a material difference in meaning? Has context-sensitive variation in recommendations been classified correctly? Have different surfaces from the same provider been merged without explanation?

Have areas outside the scope been included in the global result? Have synthetic and real distributions been separated? Does the sentence presented to the public reflect the true direction of the full distribution? If one of the substantive questions is unanswered, the distribution result carries limited confidence. If more than one is unanswered, a single global score should not be presented as a definitive judgement to the public.

30. OBJECTIONS AND ANSWERS

Objection 1 — “The public wants a single number.”

This is correct. A single number can facilitate communication. NOMOS does not completely reject a single number. However, a single number:

• scope,
• Critical error,
• subgroup base,
• time,
• uncertainty

must be connected to the information. A number can be short. It cannot shorten the truth.

Objection 2 — 'Small languages unnecessarily lower the global score.'

The global population weight can account for a small language with limited weight. However, heavy representation distortion in a small language cannot be left unnoticed simply because its population share is low. Therefore:

• population-weighted main score,
• as a result of separate language fairness

They can be published together. Two metrics answer different questions.

Objection 3 — 'If there is personalisation, different responses are normal.'

Some differences can be normal and correct. However, customisation:

• company identity,
• licence,
• its current price,
• the real service area

should not be changed arbitrarily. Personalisation can affect the user's appropriate explanation and advice. It is not permission to invent a material fact.

Objection 4 — “Why would it be a problem if the same answer is not given?”

The problem is not word difference. The problem is the change of material reality. NOMOS does not look for exact text equality, but conditional semantic consistency.

Objection 5 — “Why would it matter if a Critical error occurs only once in 1,000 answers?”

This error may be rare in terms of population prevalence. However, it can be severe in terms of damage. The uncontrolled loss of a critical system on an airplane once in a thousand flights is not assessed as only 0.1 per cent. In the context of GEO, frequency and damage should also be examined separately.

Objection 6 — “We cannot find a sufficient sample for every small country or language.”

This is true. Therefore:

• global population panel,
• Country Observer panel,
• Language Fairness Panel

can be established separately. A single-person observation does not turn into a country score. It only indicates that an observation exists. The correct status for an insufficient sample:

INSUFFICIENT SAMPLE

or:

NOT ESTIMATED

must be.

Objection 7 — "This much detail is too complex for the client to understand."

It is not necessary for the client to see all the formulas on the main page. However, the key facts necessary for the decision cannot be hidden. The short results card should answer the following questions: What is the overall result? What is the most serious risk? What is the weakest area? What has not been tested? How robust is the result? What date does it belong to? Complexity needs to be managed. Not eliminated.

Objection 8 — "If the same model is used in all countries, why would there be a country difference?"

The output seen by the user may not consist solely of the bare model. Country differences include:

• product access,
• web resources,
• locale,
• language,
• plan,
• interface,
• legislation,
• personalisation

It can emerge through it. A difference may not occur. However, its absence cannot be assumed without measurement.

Objection 9 — “Reliable scoring is not possible since AI responses are variable.”

Variability does not make measurement impossible. On the contrary, it is one of the features that needs to be measured. The score:

• specific date,
• scope,
• confidence interval,
• stability information

can be meaningful as long as it carries it. The claim of timeless certainty, however, is indefensible.

Objection 10 — “If the prompt changes, the distribution completely changes.”

That is correct. Therefore, the distribution is conditional on the prompt. GEO-1000 does not produce the single answer for all possible questions. Predefined:

• Core Mirror Prompt,
• Diagnostic Prompt Registry

Measures specific representational domains. the prompt constitution will be defined in Section 10.

COMMON RULE OF SECTION 32

This section was not written against a single score. It was written against assuming a single score alone represents the entirety of reality. People want numbers. Institutions want rankings. Boards want to see a single line. The market may ask: “What is your NOMOS score?” This question is legitimate. But the answer cannot be only: “90.” Because behind 90, any of the following realities may exist:

• Consistent accuracy across all languages
• 99 in some languages, 45 in others
• High average but severe Critical error
• High accuracy but low coverage
• High instant results but rapid decline
• Success only in easy prompts
• High rate only among assessable answers
• Assuming untested countries positively
• Combining selected AI products

The value of a score is formed by the visibility of the underlying distribution. If the distribution is not visible, the score does not generate confidence. It only produces an appearance of certainty. Therefore, NOMOS's second measurement law is as follows:

a GEO score can be a summary of the distribution. / It cannot replace the distribution.

Its third law is:

A high average does not eliminate the question of who is poorly represented.

The fourth law is as follows:

A rare serious mistake is not insignificant just because it is rare.

The fifth law is as follows:

The untested field is not the positive side of the distribution; it is the unknown boundary.

Order 2 of NOMOS

Don't just show me the average.

Show who is below the average.

Do not hide your worst language while showing your best language.

Do not use the weight of big countries to erase the existence of small countries.

Do not send one person to a small country and then announce a definitive score for that country.

Do not trivialise one severe error merely because it is hidden within a response that is 99 per cent correct.

Do not declare an error merely because the wording of an answer has changed. / But when the underlying reality changes, do not dismiss it as a mere stylistic difference.

Do not throw rejections, unknowns, and untested ones outside the distribution.

Do not change the weight, share, or scope after seeing the result.

Generate a number. / But do not make the countries, languages, users, times, and errors behind the number invisible.

Because the representation of an entity is not an average sentence. / It is a distribution of the different realities encountered by different people.

The Chapter's Closing Sentence

The measurement object of GEO is not a single answer or a single average score of an artificial intelligence; it is the entirety of representative results generated in a defined user population under defined conditions.

Normative Core

An entity's generative-system representation MUST be treated as a conditional distribution of observable outcomes across defined: - AI products and user surfaces, - users or sampling units, - countries, - languages and locales, - session and personalisation states, - prompts, - and measurement times. A single aggregate score MUST NOT replace disclosure of: - scope and coverage, - outcome distribution, - subgroup variation, - critical-error incidence, - unknown and untested mass, - temporal stability, - weighting rules, - and uncertainty. Equal averages MUST NOT be interpreted as equivalent representation quality when their subgroup, severity, coverage, or stability distributions materially differ. Untested groups MUST NOT be imputed as conforming, and rare high-harm outcomes MUST remain separately visible from average performance.