The AI System Register and Comparability

Chapter Boundary

The first eight chapters established the following: a single response is not a GEO score; representation is a distribution; the audited entity must be identified before results are observed; official representation and verified entity reality must remain separate; an eligible user population must be defined for every AI product; 1,000 valid observations must be allocated through a population-based sample design; smaller countries and languages must remain visible through complementary panels; and participant selection, exclusion and weighting must remain independent of response content. We must now define the other side of the experiment:

Which AI system are participants actually measuring?

The name of a provider? A base model? A chat product? A free web interface? A paid mobile app? A session with web search enabled? A response mode that does not cite sources? An enterprise workspace supported by in-house data? An account with memory enabled? Temporary or clean chat? Are different user interfaces of the same provider the same system? Should two products using the same model family be measured in the same way? Even if a product shows the same model name on the screen:

• retrieval,
• source selection,
• security rules,
• personalisation,
• tool access,
• experimental features

If it changes, can the results be compared? When an AI product is updated in the middle of a measurement wave, can the first and last responses of the wave be considered to come from the same system? A user's:

• web search is on,
• memory is on,
• has previous conversations,
• paid plan

responses; another user's:

• search is off,
• memory is off,
• new session,
• free plan

Can it be merged in the same cell as its answer? No. The measurement principle established by the previous sections is clear: There is no single “AI”; the search index, retrieval system, reranker, language model, citation layer, and safety system can behave differently. The same framework requires each auditable result to carry:

• model or product,
• date,
• country,
• language,
• query set,
• number of repetitions,
• measurement record

This section operates those principles within an AI System Register. This section:

• the distinction between provider, product, model, and user surface,
• AI product instance,
• model identity confidence levels,
• retrieval and tool configurations,
• memory and personalisation states,
• differences in plan, interface, and experimental features,
• product changes,
• system configuration at the time of measurement,
• levels of comparability between products,
• Native Reach and Common Support comparisons,
• version drift over time,

defines. This chapter does not yet:

• full translation and semantic equivalence system of prompts,
• all details of controlled and natural user panels,
• NOMOS Capture software architecture,
• the atomic claim scoring of the responses,
• the final model coefficients and the NOMOS scoring formula

does not finalise. The key question of Section 9 is:

Before comparing two responses from the same prompt, how do we prove that these responses actually come from comparable AI product conditions?

NOMOS Challenge

Two users of the same provider ask the same question: “What kind of company is Apple.com?” First user:

• is using the free web product,
• opens a new chat,
• has memory off,
• does not have web access,

receives an answer only with the product’s built-in knowledge capacity. Second user:

• Using a paid mobile product,
• memory is on,
• has previously talked about Apple,
• web search is on,

receives answers with current sources and citations. First answer: “Apple is a technology company that produces consumer electronics.” Second answer: “Apple is a global technology company offering hardware, software, and digital services. You can see current corporate information in these sources.” Both answers come from the same provider.

Is this the result of the same system? No. Now consider two other users. Both see the same model name on the screen. In one user’s country, the web feature is on. In the other user’s country, it is off. In one user’s answer, source links are visible. The other user’s interface does not show citations even if it uses the same sources. There is the same model label.

Is there the same measurement cell? No. Now consider two products from different providers. Both:

• new chat,
• memory is off,
• same language,
• same country,
• same time window,
• web access enabled,
• same prompt version

are measured under these conditions. They are based on different underlying models. Are they comparable? Yes, they can be comparable in terms of a specific user outcome. Because the object of comparison is:

not whether they use the same underlying model, but what representation they produce under the same defined user condition.

Now, let the product version remain unchanged for the first 500 observations of the measurement wave. During the next 500 observations, the provider:

• change the retrieval system,
• security rules,
• user interface

The report alone says: “1,000 users tested the same AI product.” In reality, two different product situations may have been measured. The first provision of this section is as follows:

The provider name is not sufficient to identify the measured AI system.

Its second provision states:

The model name is not the entirety of the product behaviour that reaches the user.

Its third provision states:

The same provider, the same model label, or the same logo does not guarantee comparability.

Its fourth provision states:

Different providers may be comparable in terms of user outcomes if the measurement conditions are sufficiently matched.

Its fifth provision states:

If a product materially changes during measurement, it cannot be reported as a single fixed system.

1. PURPOSE OF THE CHAPTER

The purpose of this section is to define the AI product audited within GEO-1000 with all its material configurations and to determine under which conditions different product outcomes can be compared. The section standardises the following distinctions:

• Provider and AI product
• AI product and base model
• Model family and specific deployment
• Model label and verified model identity
• User surface and API
• Web and mobile product
• Free plan and paid plan
• Consumer product and enterprise product
• Basic model answer and retrieval-augmented answer
• Live web search with retrieval
• Citation display with source usage
• Tool usage with tool access
• Session context with memory
• System policy with custom instruction
• Material reality with personalisation
• Temporary chat with new conversation
• Product status over time with static product name
• Observed behaviour change with provider-announced update
• Comparison of Native Reach and Common Support
• Comparable product example within the same product family
• User result versus in-model abstract ability
• Product absence versus product failure
• Technical error versus actual rejection
• Same prompt versus same experiment
• Same average versus same system behaviour
• Instant product snapshot versus long-term product performance

At the end of this section, each GEO-1000 audit should be able to answer the following questions:

Which provider's which product, which surface, which plan, and which configuration were measured?

How much was the model name seen on the screen really validated?

What were web, retrieval, citation, memory, personalisation, and tool status?

Did the product remain the same throughout the measurement?

At what comparability level were the two product results placed side by side?

Is the thing being compared the base model, the consumer product, the user surface, or the entire system chain?

2. CENTRAL NORMATIVE PROVISION

Each GEO-1000 observation must be linked not only with the provider or model name; it should be connected to a versioned AI System Register record that identifies the AI product instance at the time of measurement.

An AI product instance must carry at least the following fields to the extent relevant:

• Provider
• Product name
• User surface
• Product type
• Plan or account class
• Apparent model label
• Model ID confidence level
• Product or deployment version
• Country and locale
• Language
• Web or retrieval access
• Citation feature
• Tool access
• Memory status
• Custom instruction status
• Previous chat context
• Session type
• Personalisation status
• Security or policy mode, if known
• Experimental features
• Measurement date and UTC time
• Measurement wave
• Product change log
• Evidence and record confidence

When any of these fields materially change:

• new AI product instance,
• new experimental cell,
• new analysis version

may be required.

3. “AI SYSTEM” IS NOT A SINGLE OBJECT [K02; K03]

In this protocol, “AI system” is a general and high-level term. A real user response can arise from the combination of multiple layers. Candidate chain:

R = F(M, O, K, Q, T, P, S, U, C, L, t)

Here:

• M: base or guiding model
• O: product orchestration layer
• K: retrieval or knowledge source layer
• Q: reranking and source selection
• T: tools
• P: policy and security layer
• S: session, memory, and personalisation
• U: user account and plan
• C: country and locale
• L: language
• t: time

Response seen by the user:

R

The response visible to a user is not necessarily the output of the base model alone. The principal consumer-facing object measured by GEO-1000 is therefore:

the product behaviour presented to the user, not a model abstraction.

Controlled API or model research can also be conducted. It is not the same result as the main consumer panel.

4. SEVEN IDENTITY LAYERS OF THE AI SYSTEM

Each audited AI system must be defined across seven layers, insofar as each layer is relevant.

4.1. Provider Layer

The organisation that publishes or operates the product. Provider name:

• is not the same as product ID,
• model ID,
• user surface

A provider can offer more than one product. Within the same product, more than one model or orchestration system may be used.

4.2. Product Layer

The commercial, consumer, enterprise, or technical product accessed by the user. Example types:

• General chat product
• Search-assisted answer product
• Enterprise workspace
• Educational product
• Embedded assistant
• Developer API
• Custom knowledge-based product

The product name alone does not describe the entire configuration.

4.3. User surface Layer

It is the interface through which the user interacts with the product. Web Mobile Desktop Browser extension Operating system integration In-platform embedded interface

API

Enterprise portal Different surfaces of the same product:

• different features,
• different model,
• different source representation,
• different memory

may be used.

4.4. Model or Inference Engine Layer

It is the model, model family, or routing system used in response generation. Model ID:

• open,
• partially visible,
• unknown,
• dynamic

It is possible. A product can direct a user query to different models. In this case, one cannot say: 'A single fixed model was measured.'

4.5. Orchestration and Information Layer

It may include the following:

• Retrieval
• Live web search
• Internal search
• Source ranking
• Reranking
• Citation selection
• Calling a vehicle
• Query rewriting
• Document shredding
• Answer merging

This layer is of fundamental importance for the representation of GEO. Because it can affect which record:

• is accessed,
• is selected,
• is carried into the response,
• is shown as a reference

can be influenced.

4.6. Policy and Personalisation Layer

It may include the following:

• Security rules
• System instructions
• Product policy
• Memory
• Special instructions
• User history
• Regional rules
• Age or account class
• Corporate administrator settings

This layer affects the:

• scope of the response,
• its tone,
• its rejection,
• its recommendation

can change.

4.7. Distribution and Time Layer

Indicates which of the following the product is under at a specific moment:

• is meaningful with
• experimental feature,
• regional distribution,
• user segment,
• A/B test

The same product name can have different distribution conditions for different users on the same day.

5. AI PRODUCT EXAMPLE

The core system unit within GEO-1000:

AI Product Instance

Canonical operational term:

AI Product Instance

An AI Product Instance is defined as:

A_i = (v, p, s, r, m, k, t, x, u, c, l, ω)

can be represented as. Here:

• v: provider
• p: product
• s: user surface
• r: plan or account class
• m: model ID or label
• k: retrieval and information structuring
• t: tool configuration
• x: memory, personalisation, and session state
• u: user or organisational setting class
• c: country and locale
• l: language
• ω: time, deployment, and experimental feature state

An observation: Oj must be linked to a specific: Ai product sample.

6. DIFFERENCE BETWEEN PRODUCT SAMPLE AND PRODUCT NAME

The product name can be fixed. The product sample may vary. Example: Let the Orion AI product name remain unchanged. However, these two measurements are different product samples:

Product Sample 1

Web Free plan Memory off Web access off New chat Turkish Turkey Wave 1

Product Sample 2

Mobile Paid plan Memory on Web access on Previous chat context available Turkish Turkey Wave 1 These can exist under the same brand. They are not the same experimental cell.

7. MODEL IDENTITY CONFIDENCE LEVELS

The model name cannot always be known with the same certainty. Therefore, the model identity can be recorded at the following levels.

MI-0 — UNKNOWN MODEL

It is unknown which model or model family the product uses. The result is attributed only to the product surface.

MI-1 — USER-DISPLAYED LABEL

A model label is visible in the user interface. The label is recorded. It is not assumed to fully verify the back-end deployment.

MI-2 — PROVIDER-DOCUMENTED MODEL FAMILY

The provider's product documents describe a specific model family or product–model relationship. It may still be unknown whether individual observation is directed exactly to that model.

MI-3 — SESSION-CONFIRMED DEPLOYMENT

The session verifies a specific distribution of API or trusted product metadata.

MI-4 — VERSIONED ENDPOINT OR DEPLOYMENT

A specific version, endpoint, or deployment ID can be recorded. It is stronger for controlled comparison.

MI-5 — REPRODUCIBLE FROZEN MODEL ARTEFACT

There is a static and reproducible model or deployment structure. It may be rare in live consumer products. If model ID trust is low, the result:

should be attributed to product behaviour

It cannot be definitively attributed to a specific model.

8. LIMIT OF THE APPARENT MODEL NAME

Seeing the model name in a user interface is valuable. However, the following questions may still remain open: Are all queries directed to the same model? Does the model change depending on the difficulty of the query? Does security or retrieval use a separate model? Does the provider apply the same label to different distributions? Does the model label only indicate the product family? Is the user part of an experimental rollout? Therefore, the correct reporting might be: “Model X label was shown in the user interface.” Without stronger evidence, the following sentence should not be used: “All responses are definitively generated solely by Model X.”

9. PRODUCT TYPE

Each AI product should be classified with a primary type and secondary types.

PT-1 — General Conversational Product

It is a general chat and question-answer product.

PT-2 — Search- or Retrieval-Grounded Product

Search or retrieval is a fundamental component in answer generation.

PT-3 — Enterprise Knowledge Product

It can use internal documents, workspaces, or special data sources.

PT-4 — Embedded Assistant

It is embedded into another product or platform.

PT-5 — Developer or API Product

It is used by a developer or application instead of the end user.

PT-6 — Domain-Specific Product

It is designed for a specific profession or industry.

PT-7 — Multimodal Product

Inputs other than text, such as visual, audio, or document entries, are part of the main experience. A product can carry more than one type. Comparison should be made within the same type or with an explicit scope.

10. USER SURFACE REGISTER

Different surfaces of the same product should be recorded separately. Surface areas:

• Surface ID
• Web/mobile/desktop/API
• Application version
• Operating system, if applicable
• Browser, if applicable
• Product language
• Visible model selection
• Availability of tools
• Citation view
• File upload feature
• Memory settings
• Corporate management settings
• Experimental features
• Measurement time

If a web and mobile interface are found to be materially the same, the results can be combined. This equality should not be assumed; it must be tested.

11. PLAN AND ACCOUNT CLASS

The plan type may have features that can affect the user's response. Plan statuses:

• Free
• Paid individual
• Professional
• Corporate
• Education
• Developer
• Trial
• Invitation or limited access
• Unknown

Plans may differ in the following areas:

• Model
• Usage limit
• Web access
• Memory
• Tools
• Source citation
• Response length
• Speed or priority
• Enterprise data
• Security and retention settings

If there is no plan information:

The PLAN_UNKNOWN

status should be used. Free and paid results cannot silently be merged as the same product behaviour.

12. RETRIEVAL AND WEB ACCESS

An AI response can be generated in one of three basic knowledge states.

KG-0 — UNDETERMINED KNOWLEDGE MODE

The case of retrieval or web usage cannot be determined.

KG-1 — MODEL-INTERNAL RESPONSE

A response known or strongly assumed not to use live or external information.

KG-2 — RETRIEVAL-ASSISTED RESPONSE

Retrieval may have been used from pre-indexed or in-product sources.

KG-3 — LIVE WEB OR SEARCH-ASSISTED RESPONSE

Live or near-measurement time web/search access has been used.

KG-4 — PRIVATE KNOWLEDGE-GROUNDED RESPONSE

A private data source provided by the institution or the user has been used.

KG-5 — HYBRID OR MULTI-SOURCE RESPONSE

Multiple layers of information have been used together. These modes may produce different results in the same prompt. They should be clearly distinguished in comparison.

13. RETRIEVAL ACCESS IS NOT THE SAME AS RETRIEVAL USAGE

The product may have a web search feature. However, it may not have been used in the specific answer. Three fields should be recorded separately: Was the feature available? Was the feature on or selectable? Was there evidence that it was used in the specific answer? Example statuses:

AVAILABLE_NOT_USED

AVAILABLE_USAGE_UNKNOWN

USED_CONFIRMED

NOT_AVAILABLE

DISABLED

UNKNOWN

The mere presence of a source link can be a strong indicator for usage. The absence of a source link does not necessarily prove that retrieval was not used.

14. CITATION LAYER

Using a source is not the same as showing a citation. A product:

• may use a source but may not show it,
• may show a citation but the citation may not support the claim,
• may link the same source to different sentences,

The user can hide the citation according to the user surface. The citation record must contain the following fields: Is the citation feature available? Was the citation shown in the specific response? Number of citations Source field names Citation–claim match Visibility on the user surface Expandability of the citation Accessibility of the source Citation generation time Products that show and do not show citations cannot be directly compared in terms of "number of sources". First, product behaviour and interface differences must be explained.

15. TOOL ACCESS

The AI product may access the following tools:

• Web search
• Browser
• Calculator
• Code execution
• File reading
• Map
• Calendar
• Corporate knowledge base
• Product catalogue
• Other APIs

Tool access may affect your response:

• its timeliness,
• evidence,
• scope of the response,
• accuracy

Tool registry:

• whether the tool is available,
• whether it is active,
• whether it was used in the response,
• whether the tool result is retained

should be shown.

16. MEMORY, SPECIAL INSTRUCTIONS, AND PERSONALISATION

The response visible to the user may be affected by the following fields:

• Persistent memory
• Chat history
• Special instructions
• User's name or profile
• Previous brand conversations
• Preferences
• Country and locale
• Corporate workspace settings
• Segmentation done by the provider

These fields must be recorded in three separate layers.

16.1. Persistent Personalisation

Information protected across accounts or sessions.

16.2. In-Conversation Context

Previous messages within the current chat.

16.3. System or Organisational Setting

Administrator or product instructions that the user may not see. Complete information may not always be available. Unknown situations must be explicitly recorded.

17. SESSION TYPES

Candidate session statuses:

ST-1 — NEW STANDARD CONVERSATION

New standard conversation. Memory or special instructions are recorded separately.

ST-2 — TEMPORARY OR EPHEMERAL CONVERSATION

Temporary session intended to be separated from permanent history. Actual features should be verified in the product registry.

ST-3 — EXISTING CONVERSATION

There is previous conversation context. It may be valuable for the natural panel. It is kept separate in the controlled panel.

ST-4 — ENTERPRISE OR SHARED WORKSPACE SESSION

There may be an enterprise or shared information area.

ST-5 — UNKNOWN SESSION STATE

The session status could not be reliably determined. If different session types are to be combined in the same cell, the rationale and weighting must be explained.

18. PRODUCT POLICY AND REJECTION

An AI product may not respond to a specific question due to:

• security,
• law,
• product policy,
• regional rule,
• does not imply any suitability in terms of age,
• enterprise setting

reason. Rejection:

• basic model deficiency,
• product policy,
• interface,
• may be due to the user account

GEO-1000 main user control records the rejection result that reaches the user. However, if the reason for the rejection is known, it is additionally classified. A product's policy layer is also part of the user experience.

19. EXPERIMENTAL FEATURES AND A/B DISTRIBUTIONS

Live AI products may not give the same features to all users at the same time. Participants:

• different user interface,
• different model prompting,
• different web feature,
• different response format

may see. Experimental status indicators:

• Beta label on the interface
• Provider documentation
• Feature difference
• Systematic UI difference between users
• Behaviour breaking during measurement

may occur. If the experimental feature is unknown:

EXPERIMENTAL_STATE_UNKNOWN

should be used. The actual variation within the product is not data defect. It is a distribution feature of the system. However, it should be considered in comparability.

20. AI System Register

An AI System Register should be created for each audited product. The registry should carry three levels.

20.1. Provider and Product Master Record

Includes relatively permanent areas:

• Provider
• Product name
• Product type
• Main user interfaces
• Official access countries
• Plan classes
• Supported language records
• General features

20.2. Measurement Wave Product Record

For a specific wave:

• product version,
• plan features,
• model labels,
• web and tool features,
• memory behaviour,
• access status,
• known updates

is recorded.

20.3. Observation Level System Record

Actually observed for each participant:

• interface,
• model label,
• plan,
• vehicle usage,
• citation,
• session,
• personalisation,
• time

carries fields. The main record alone cannot remove observation-level differences.

21. SYSTEM CONFIGURATION FINGERPRINT

A System Configuration Fingerprint can be created to facilitate the comparability of an AI product instance. Candidate fields:

SCF = H(canonicalise(p, s, r, m, k, t, x, c, l, w))

Here, H is the function that produces an integrity value from the versioned fields. The fingerprint:

• whether two observations belong to the same configuration,
• whether there have been material changes in the wave

It can help to identify. Fingerprint: does not prove the real backend model, does not make layers that the provider has not disclosed visible. It only reinforces whether the recorded configuration has been changed later.

22. WHAT IS COMPARABILITY?

Two results are considered comparable when:

The difference between them can be sufficiently explained by monitored product behaviour rather than irrelevant differences in measurement design.

Comparability is not absolute equality. Different AI products:

• model,
• provider,
• user surface.

These need not all be identical. The material conditions relevant to the comparison must, however, be:

• equalisation,
• layering,
• recording,
• modelling in analysis

are required.

23. THE OBJECT OF THE COMPARISON MUST BE DEFINED IN ADVANCE

The same data can answer different comparison questions.

23.1. User Product Comparison

"Which consumer product produces a more accurate representation under the same user conditions?" The object being compared is the entire product chain.

23.2. Model Comparison

"Which model produces a more accurate response under the same controlled prompt and tool-free condition?" API or fixed model access may be required.

23.3. Retrieval Comparison

“Is there a difference between the web-accessible and non-accessible modes of the same product?” The model and other conditions are kept as constant as possible.

23.4. Plan Comparison

“Do free and paid users see the same brand representation?” The plan difference is the object of the audit.

23.5. Interface Comparison

“Does the web and mobile product produce different representations under the same conditions?”

23.6. Time Comparison

“Did the representation of the same product change across different waves?” Product and population changes should be separated. Without specifying the comparison question: “Product A is better than Product B.” is excessively broad.

24. DIMENSIONS OF COMPARABILITY

When comparing two products or waves, the following areas should be evaluated:

• Audited entity
• Reality package version
• Prompt ID and version
• Language and locale
• Country
• Target population
• Sample source
• Participant composition
• Session type
• Memory
• Special instructions
• Previous conversation
• Product plan
• User surface
• Model label
• Retrieval/web status
• Tools
• Citation feature
• Measurement time
• Product updates
• Adjudication rule
• Result statuses
• Weighting

Not all dimensions need to be the same. Different dimensions limit the scope of the comparison.

25. COMPARABILITY LEVELS

CP-0 — INCOMPARABLE

Material measurement conditions are unknown or seriously inconsistent. Direct performance comparison cannot be made.

CP-1 — DESCRIPTIVE CO-OCCURRENCE

There are results from two products or waves. However:

• population,
• prompt,
• time,
• session

are not equal. Only descriptive side-by-side display can be made.

CP-2 — PARTIALLY ALIGNED

Some core fields match. Material differences continue. Limited comparison can be made.

CP-3 — MATCHED USER-SURFACE COMPARISON

Same entity Same prompt version Same language/country Similar user population Same time window Registered product features are available. Consumer product results are comparable.

CP-4 — CONTROLLED MATCHED COMPARISON

Additionally:

• session,
• memory,
• personalisation,
• retrieval,
• tools,
• plan

are matched in a controlled or open manner. Provides stronger product or feature comparison.

CP-5 — REPLICATED AND CALIBRATED COMPARISON

Comparison:

• in multiple waves,
• with independent users or panels,
• with the same normative evaluation

repeated; variability and product changes have been calibrated. These levels are not a “scientific accuracy score.” They indicate the limit that a comparison claim can carry.

26. WHAT TO DO IN CP-0 CONDITION?

If two products cannot be compared, the results are not ignored. They can be reported separately:

• “This is the Native Reach result of Product A.”
• “This is the result of Product B under different users and product conditions.”

However: one cannot say, “A is 12 points better than B.” Incomparability is not a failure, it is a limitation of the method.

27. COMPARABILITY FOR THE SAME PROVIDER

Results from the same provider cannot be automatically compared. The following differences may be present:

• Product
• Plan
• Model
• Web access
• Interface
• Memory
• Enterprise data
• Country distribution
• Experimental feature
• Time

Comparison of the same provider should also be evaluated according to the CP level.

28. COMPARABILITY FOR DIFFERENT PROVIDERS

User products belonging to different providers:

• same target population,
• same prompt,
• same time,
• similar user surface,
• clear product configuration

It is comparable underneath. Being a different provider does not prevent the comparison. However, differences in features determine what the comparison measures. Example: Product A uses live web. Product B does not. The result can answer this question: 'Which of the two product chains offered to the user produces more accurate representation?' It does not directly answer this question: 'Which base model is more knowledgeable?'

29. COMPARISON OF WEB-SUPPORTED AND NON-WEB PRODUCTS

Two separate comparisons are possible.

29.1. Natural Product Comparison

Each product is measured under the default real user condition. A product that uses the web uses the web. One that does not use it does not. Result: it is a comparison of the real consumer product experience.

29.2. Feature Matched Comparison

If possible, both products:

• measured in web open,
• web closed

modes separately as well. This can better separate the retrieval effect. Each product may not offer the same options. Missing mode:

RECORDED AS NOT AVAILABLE

That status must remain explicit.

30. PRODUCTS THAT SHOW AND DO NOT SHOW CITATIONS

For a product without citation feature: it is possible to give a “Zero citation.” However, its meaning should be explained: the product may not be referencing sources. It is not certain that it does not use any sources. Citation coverage and accuracy coverage are separate outcomes. Citation comparison should only be made among products that support citations or with separate feature status.

31. NATIVE REACH COMPARISON

Each product is measured within its actual reach population. Advantage: shows the product’s natural global impact. Limitation: populations are different, country and language distributions may vary. Native Reach scores: show what the product does within its reached population. Should not be used alone in direct product superiority ranking.

32. COMMON SUPPORT COMPARISON

The common access population of the compared products is used. Advantage:

• same country,
• same language,
• same user condition

provides a fairer comparison. Limitation: preserves the actual global access difference of the products, the common universe may be very narrow. The Common Support score should be published together with the Native Reach score.

33. PAIRED USER COMPARISON

If the same user tests two or more AI products, user characteristics can be controlled. However:

• task order,
• answer recall,
• product familiarity,
• special profile of users who can access all products

It can create bias. In a paired design:

• product order should be randomised,
• previous responses should not be shared,

the same user dependency should be maintained in the analysis.

34. INDEPENDENT USER COMPARISON

Each product is tested by different users, but selected from the same population frame. Advantage: task contamination is reduced, more natural product users can be chosen. Limitation: user groups may not be completely equal. Weighting and stratification are required. Paired and independent user results can be separate but complementary.

35. TIME COMPARABILITY

When comparing today's result of an AI product with past results, the following changes should be examined:

• Model
• Product version
• Web/retrieval
• Plan
• Country access
• Language support
• Prompt
• Entity reality
• Adjudication
• Population framework
• Weighting

Time difference may not only arise from the AI product.

36. TYPES OF PRODUCT CHANGES

UC-1 — DOCUMENTED MINOR CHANGE

A minor change disclosed by the provider and assessed not to materially alter the measurement object.

UC-2 — DOCUMENTED MATERIAL CHANGE

Material change described in model, retrieval, interface, or feature.

UC-3 — OBSERVED BEHAVIOUR CHANGE

Even if there is no official announcement, systematic behaviour or interface change is observed.

UC-4 — ACCESS OR PLAN CHANGE

Country, plan, or user access changes.

UC-5 — UNKNOWN CHANGE EVENT

There is a break in measurement behaviour. The reason could not be verified.

UC-6 — PRODUCT REPLACEMENT OR RETIREMENT

An old product or model has ended and has been replaced by another structure. The type of change limits time series comparison.

37. PRODUCT CHANGE WITHIN THE MEASUREMENT WAVE

If a material change occurs during a wave, the following options exist: The wave can be stopped and restarted. Separate sub-waves can be made before and after the change. The change can be modelled as an experimental factor. The wave is labelled as MIXED_SYSTEM_STATE. The following structure is forbidden: Combining all responses as if they result in a single fixed product outcome despite knowing about the change.

38. BEHAVIOUR CAN CHANGE WITHOUT PRODUCT CHANGE

Even if the provider does not announce an explicit update, the result may change. Possible reasons:

• Change in the retrieval index
• Updating of resources
• New content appearing on the web
• Reranking change
• A/B test
• Regional rollout
• Security policy
• Traffic routing

Therefore, it cannot be said, “The provider did not update, the system was definitely the same.” Measurement evidence and behaviour breakdown should also be examined.

39. LIVE PRODUCT AND FROZEN MODEL

There are two different research objects.

39.1. Live Product Audit

Measures what real users see on the real consumer surface. Advantage:

• shows real experience,
• real source,
• real policy,
• real interface

limitations: it is variable, full reproduction may be difficult.

39.2. Frozen Model Audit

Controlled repetition is conducted under a fixed version or endpoint. The advantage:

• provides reproducibility,
• experimental control

The limitation: may not fully represent the real consumer product. The main population layer of NOMOS GEO-1000 is based on the live user product. Frozen model audit is complementary laboratory work.

40. PRODUCT UNAVAILABILITY AND TECHNICAL RESULTS

The following cases should be kept separate:

AR-1 — PRODUCT UNAVAILABLE

The product is not available for the country or user group. It is the result of access scope.

AR-2 — ACCOUNT OR PLAN INELIGIBLE

The plan or account condition assigned to the participant is not eligible. There may be a sampling or task assignment issue.

AR-3 — TEMPORARY TECHNICAL FAILURE

A temporary technical error has occurred. Retry protocol should be predefined.

AR-4 — RATE OR USAGE LIMIT

A usage quota or rate limit has been reached. It may be part of the real user experience.

AR-5 — SYSTEM REFUSAL

The product task has been refused due to policy or content. It is a valid system outcome.

AR-6 — NO USABLE OUTPUT

No usable response has been generated.

AR-7 — UNKNOWN ACCESS FAILURE

Why it could not be determined. These situations cannot be combined in the same "no response" box.

41. TECHNICAL RETRY RULE

A retry can be made in case of a temporary technical error or connection interruption. However:

• if the AI response is not liked,
• a rejection occurred,
• incorrect information was produced

retry cannot be made for these reasons. Technical retry:

• same user,
• same product,
• same configuration,
• must be made within the predetermined period.

The first technical incident must also be recorded.

42. COMPARISON MATRIX

A basic comparison record for two products could be as follows:

These results can be shown side by side. However: "The basic model of Product B is better." cannot be said. The difference may stem from the entire product chain.

43. SYNTHETIC APPLE.COM COMPARISON CASE

SYNTHETIC METHODOLOGY DEMONSTRATION / The products, responses, and results below are entirely fictional. They do not represent the performance of the real Apple Inc. or any real AI provider. Let's consider three synthetic product examples.

Product Example A

Provider: Orion Product: Orion Chat Interface: Web Plan: Free Model label: Orion Standard Model Confidence: MI-1 Web access: Off Memory: Off Session: New Language: Turkish Locale: Turkey Wave: W1 Response: "Apple is an American technology company that produces phones and computers."

Product Sample B

Provider: Orion Product: Orion Chat Interface: Mobile Plan: Paid Model Label: Orion Pro Model Confidence: MI-1 Web Access: Open Citation: Open Memory: On Previous Conversation: About Apple Language: Turkish Locale: Turkey Wave: W1 Response: "Apple is a global technology company that provides hardware, software, and digital services. Current corporate information can be found in these sources."

Product Sample C

Provider: Helios Product: Helios Search Interface: Web Plan: Free Model Label: Unknown Model Confidence: MI-0 Web Access: Open Citation: Open Memory: None Session: New Language: Turkish Locale: Turkey Wave: W1 Response: "Apple is a US-based technology company that develops consumer devices, operating systems, and digital services."

43.1. Comparison of A and B

They are the same provider. However:

• plan,
• surface,
• model label,
• web,
• memory,
• previous conversation

is different. The comparison level can be CP-1 or CP-2. The response difference cannot be attributed solely to the basic model difference.

43.2. Comparison of A and C

They are different providers. However:

• new session,
• free plan,
• web surface,
• same language/country,
• same wave

in terms of some conditions match. Web access is different. Comparison: can be done as a natural consumer product outcome. It cannot be done as a basic model capability.

43.3. Comparison of B and C

Both products have web access and citations enabled. However, B has memory and previous conversation, while C does not. The user experience of the products is comparable. It is not a controlled clean product comparison.

43.4. Controlled Repeat

In product B:

• memory is turned off,
• a new chat is started,
• special instructions are turned off,

web remains on. C is measured under the same condition. Comparison level: CP-3, or CP-4 with sufficient additional control. This example shows:

The same provider does not provide a close comparison; well-recorded different providers may be more comparable.

44. SYNTHETIC PRODUCT CHANGE CASE

SYNTHETIC EXAMPLE

Orion Chat W1 wave targets 1,000 observations. In the first 480 observations:

• product version: 6.1
• web feature: manual
• attribution: limited

In the next 520 observations:

• product version: 6.2
• web feature: automatic
• attribution: extended

let it be. Single result: if calculated as “Orion Chat W1 score,” two different system states mix. Correct options:

Option 1

W1A: first 480 W1B: next 520 are reported separately.

Option 2

The wave is not continued. 1,000 new observations are taken in version 6.2.

Option 3

The version is kept as an explicit experimental factor in statistical analysis. Incorrect option: The update is removed from the report and all observations are presented as a single fixed product.

45. MINIMUM AI SYSTEM RECORD

Each observation must carry at least the following fields:

• NOMOS AI system ID
• Provider
• Product
• Surface
• Plan
• Product type
• Apparent model label
• Model confidence level
• Country
• Locale
• Language
• Session type
• Memory status
• Custom instruction status
• Previous conversation state
• Retrieval/web feature
• Retrieval usage status
• Citation status
• Tool status
• Experimental feature status
• UTC time
• Wave
• System fingerprint
• Change events
• Source of evidence
• Record owner

If the material field is unknown, it cannot be left blank. UNKNOWN status should be used.

46. MINIMUM COMPARISON CARD

When comparing two products or waves, the public record should show:

• Comparison question
• Comparison level
• Common target population
• Native Reach differences
• User equality
• Language and country
• Measurement time
• Product, plan, and surface
• Model identity confidence
• Web/retrieval difference
• Difference in memory and personalisation
• Difference in tool
• Difference in sample
• Evaluation method
• Material limitations
• Directly comparable metrics
• Non-comparable areas

Single number or ranking should not be published without this card.

47. RULE OF COMPARABILITY

An auditor should answer these four questions: Was the same thing asked? Was the same human universe measured? Were the same user conditions applied? Were the same time and evaluation system used? Each of these questions:

• completely,
• partially,
• no,
• unknown

is classified as. The comparison level is given accordingly.

48. MANDATORY NORMATIVE PROVISIONS

CH09-N01

Every GEO-1000 observation must be linked to a versioned AI System Register record.

CH09-N02

The provider name cannot be used as the full identity of the measured AI product.

CH09-N03

The product, user surface, plan, model, retrieval, tool, session, and personalisation layers must be separated from each other.

CH09-N04

The model label on the user surface cannot be presented as a verified back-end model identity.

CH09-N05

The model-identity confidence level must be recorded from MI-0 to MI-5 or through an equivalent openly defined status.

CH09-N06

If the model identity cannot be verified, the result should be attributed only to the product and user surface.

CH09-N07

Products belonging to the same provider are not automatically considered the same experimental cell.

CH09-N08

Products belonging to different providers may be comparable when material user and experimental conditions are matched.

CH09-N09

Web, mobile, desktop, embedded product and API surfaces cannot be combined without description.

CH09-N10

Free, paid, corporate, educational, and developer plans must be recorded as separate product terms.

CH09-N11

If the plan is unknown, the result should carry the status PLAN_UNKNOWN.

CH09-N12

The availability of the retrieval or web feature cannot be counted as proof that it was used in a specific response.

CH09-N13

Retrieval access, retrieval usage, and attribution must be recorded with separate statuses.

CH09-N14

A product that does not show attribution cannot automatically be considered as not using any resources.

CH09-N15

The citations of the product being cited should be examined separately in terms of claim support.

CH09-N16

Tool access and tool use in specific responses should be recorded separately.

CH09-N17

Memory, special instructions, previous chats, and corporate settings should be recorded separately as much as possible.

CH09-N18

An unknown personalisation state cannot be assumed to be like a clean session.

CH09-N19

New chat, temporary chat, current chat, and corporate workspace sessions cannot be merged without explanation.

CH09-N20

Natural user panel and controlled clean panel results should carry separate product status.

CH09-N21

If an experimental feature or A/B distribution in the product is material, it should be preserved in the observation record.

CH09-N22

If a material product change occurs during a wave, the wave should be divided into separate sub-versions, restarted, or the change should be explicitly modelled in the analysis.

CH09-N23

A known product change cannot be stored as if all observations came from a single fixed system.

CH09-N24

The provider not explaining an update cannot be taken as definitive proof that the product has not changed.

CH09-N25

The result of a live consumer product cannot be presented as if it were the result of a frozen baseline model.

CH09-N26

The result of a frozen model in the laboratory cannot be presented as performance of a real user product.

CH09-N27

Absence of the product, technical error, usage limit, system rejection, and lack of result must be recorded with separate statuses.

CH09-N28

Incorrect or negative AI response cannot be a reason for technical retry.

CH09-N29

The subject of the comparison must be defined before data is collected.

CH09-N30

The comparability level of two products must be visible in the public result.

CH09-N31

Definite product ranking cannot be generated from CP-0 or CP-1 level results.

CH09-N32

Using the same prompt alone does not create a comparable experiment.

CH09-N33

In comparisons, the target entity, Truth Pack, prompt, language, country, time, population, session, product, and evaluation method should be examined together.

CH09-N34

Native Reach results cannot be directly ranked without explanation because they belong to different target populations.

CH09-N35

The Common Support result cannot be presented as the product's actual global reach result.

CH09-N36

Web-supported and web-free product comparison can be done as a user product comparison; it cannot be presented as a basic model superiority.

CH09-N37

In paired comparisons made with the same user, product order and user dependency must be maintained.

CH09-N38

In comparisons made with different users, user composition and weight differences must be visible.

CH09-N39

The product version or configuration fingerprint must be stored in every measurement wave.

CH09-N40

There must be an accountable human or institutional owner of the system registry and comparability decision.

49. FORMS OF FAILURE

CH09-F01 — CONSIDERING THE PROVIDER AS A SINGLE AI

All products, models, and surfaces of the provider are presented as a single system.

CH09-F02 — COUNT PRODUCT BEHAVIOUR BY MODEL NAME

Retrieval, security, interface, and personalisation are ignored.

CH09-F03 — DEFINITELY CONSIDER THE VISIBLE MODEL LABEL AS BACKEND

The use of routing or dynamic models is disregarded.

CH09-F04 — CONSIDERING THE SAME PROVIDER AS COMPARABLE

The plan, surface, and feature differences are hidden.

CH09-F05 — CONSIDERING DIFFERENT PROVIDERS AS NON-COMPARABLE

Product comparisons possible under paired user conditions are rejected.

CH09-F06 — INTEGRATING WEB AND MOBILE

Interface and feature differences disappear.

CH09-F07 — COMBINING FREE AND PAID PLANS

Quality or feature differences due to the plan become invisible.

CH09-F08 — CONSIDERING CORPORATE AND CONSUMER PRODUCTS THE SAME

Private information and administrator settings are hidden.

CH09-F09 — TREATING API RESULT AS CONSUMER PRODUCT

Orchestration and user surface differences are eliminated.

CH09-F10 — IF RETRIEVAL EXISTS, CONSIDER IT USED

The specific answer's true knowledge path cannot be verified.

CH09-F11 — IF NO CITATION, CONSIDER NO SOURCE

The possibility of hidden or invisible retrieval is ignored.

CH09-F12 — IF CITATION EXISTS, COUNT AS CORRECT SOURCE

Citation–claim matching is not examined.

CH09-F13 — CONSIDER TOOL ACCESS AS TOOL USE

It is unknown which tool was used in the specific answer.

CH09-F14 — CONSIDER MEMORY AS CLEAN SESSION

The effect of personalisation remains invisible.

CH09-F15 — CONSIDER NEW CONVERSATION NON-PERSONALISED

Persistent account settings and special instructions are ignored.

CH09-F16 — PRESENTING THE CURRENT CHAT AS A NEW SESSION

Previous context is hidden.

CH09-F17 — COUNTING AN EXPERIMENTAL FEATURE AS A DATA ERROR

Actual product variation is declared an invalid observation.

CH09-F18 — HIDING AN EXPERIMENTAL FEATURE

Users in different rollouts are combined as a single fixed product.

CH09-F19 — SINGLE WAVE AFTER UPDATE

Material product change is published as the result of a single fixed system.

CH09-F20 — IGNORING CHANGES IF THERE IS NO ANNOUNCEMENT

Observed behaviour break is not examined.

CH09-F21 — COUNTING LIVE PRODUCT AS A FROZEN MODEL

Full reproducibility is claimed.

CH09-F22 — CONSIDERING A FROZEN MODEL AS A REAL USER PRODUCT

Product chain and interface differences are hidden.

CH09-F23 — COUNTING PRODUCT ABSENCE AS AN ERROR

Scope of access is confused with representation accuracy.

CH09-F24 — REFUSING A TECHNICAL ERROR

Connection or service issue is classified as a policy result.

CH09-F25 — COUNTING A TECHNICAL ERROR AS REJECT AND RETRYING

Actual user result is removed from the denominator.

CH09-F26 — HIDING THE USAGE LIMIT

The real experience of the free plan becomes invisible.

CH09-F27 — CONSIDERING THE SAME PROMPT AS THE SAME EXPERIMENT

Product, plan, session, web, and time differences are ignored.

CH09-F28 — CONSIDERING THE PRODUCT RESULT AS MODEL SUPERIORITY

The difference across the entire system chain is attributed to the base model.

CH09-F29 — DIRECTLY RANKING NATIVE REACH SCORES

Different population universes are considered the same.

CH09-F30 — CONSIDERING COMMON SUPPORT AS GLOBAL SCOPE

The narrow common universe is presented as the entire reach of the product.

CH09-F31 — CLAIMING LEADERSHIP FROM CP-1 RESULT

Product superiority is derived solely from incompatible results that are placed side by side.

CH09-F32 — MAINTAINING THE COMPARISON LEVEL

The public only sees the ranking.

CH09-F33 — IGNORING PAIRED USER DEPENDENCY

The responses of the same person to products are considered independent.

CH09-F34 — IGNORING PRODUCT ORDER EFFECT

The first response affects subsequent tasks.

CH09-F35 — TREATING DIFFERENT PANEL USERS AS THE SAME

Products are tested with different user compositions.

CH09-F36 — CHANGING THE REALITY PACKAGE VERSION

The same responses are compared with different reference records.

CH09-F37 — SILENTLY CHANGING THE PRODUCT FINGERPRINT

The configuration record is updated; the old measurement is preserved rather than overwritten.

CH09-F38 — COUNTING AN UNKNOWN AREA AS CLEAN

The model, memory, or tool status is accepted as controlled, even if unknown.

CH09-F39 — USING THE PRODUCT BRAND INSTEAD OF EVIDENCE

The provider's reputation substitutes for measurement conditions.

CH09-F40 — COVERING ALL PRODUCT DIFFERENCES WITH A SINGLE COEFFICIENT

Plan, interface, and configuration differences are hidden by using the model or market coefficient.

50. AUDIT PROCEDURE

Step 1 — Identify the Provider and Product

The product name, type, and official user interfaces are recorded.

Step 2 — Lock the Measurement Surface

Web Mobile Desktop

API

Embedded product is detached.

Step 3 — Verify the Plan and Account Class

Free, paid, enterprise, or other plan status is recorded.

Step 4 — Assign Model ID Trust

Appropriate level is determined between MI-0 and MI-5.

Step 5 — Save Retrieval and Web Configuration

Is the feature available? Is it on? Was it used in a particular response? What is the evidence?

Step 6 — Save Reference and Tool Status

Actual usage is separated from feature access.

Step 7 — Save Session and Personalisation Status

Memory Special instruction Previous chat Corporate setting is reviewed.

Step 8 — Verify Country, Locale, and Language Registration

The regional condition of the product behaviour is determined.

Step 9 — Review Experimental Features

UI or feature differences between users are checked.

Step 10 — Lock Time and Wave Identity

UTC time and wave record are created.

Step 11 — Generate System Configuration Fingerprint

Saved fields are linked to the integrity record.

Step 12 — Monitor Change Events

Announced and observed product changes are recorded.

Step 13 — Write the Object of Comparison

Which of the Product Model Retrieval Plan Interface Time comparisons is being made?

Step 14 — Match the Comparison Dimensions

Material areas are tabulated.

Step 15 — Assign CP Level

An appropriate level is given between CP-0 and CP-5.

Step 16 — Separate Comparable and Non-Comparable Claims

Which results can be placed side by side is clearly written.

Step 17 — Separate the Results of Native Reach and Common Support

Intra-product and common population comparisons are recorded separately.

Step 18 — Check System Change in the Wave

It is determined whether MIXED_SYSTEM_STATE or a new sub-wave is needed.

Step 19 — Create Public Comparison Card

The conditions behind the ranking are made visible.

Step 20 — Obtain Record Owner's Approval

The system registry and comparison decision are versioned by the responsible person.

51. REQUIRED EVIDENCE

Provider identity; product name and type; user surface; application or interface version; plan and account class; displayed model label; model-identity evidence; model-identity confidence level; product documentation; country-access records; language and locale; web and retrieval capability; evidence of retrieval use; citation capability; citation view; citation–claim link; tool access; tool-use log; memory setting; custom instructions; prior-chat state; session type; organisational administrator settings; experimental-feature record; A/B allocation signals; UTC timestamp; measurement wave; system-configuration fingerprint; provider-update records; observed behaviour changes; product-access errors; technical-retry logs; Native Reach population record; Common Support population record; and comparison-dimension table.

CP level Prompt version Reality package version Sample and weight record Adjudication version Public comparison card Change log Responsible person or institution

52. AUDIT CHECKLIST

Were the provider and product recorded separately? Is the user surface identifiable? Were the plan and account class verified? Is the model label merely the displayed label, or does it identify a verified deployment? Was a model-identity confidence level assigned? Is the result attributed to the product or to the model? Was web or retrieval available? Was its use in the specific response verified? Were citation capability and citation display separated? Do the citations actually support the claims? Were tool access and tool use separated? Was memory enabled? Were custom instructions present? Was there prior conversational context? Was the session type recorded? Was an enterprise or private information source present? Were experimental product features checked? Could country and locale affect product behaviour?

Has the system fingerprint been created? Did the product change during the measurement wave? Is there any behaviour break outside the provider announcement? Were pre-change and post-change observations separated? Were technical errors distinguished from rejections? Was a retry performed due to an incorrect answer? Is the object of the comparison clear? Which conditions match besides the same prompt? Is the target population the same? Are the prompt and Truth Pack versions the same? Are the differences in plan, surface, and retrieval visible? Was the CP level provided? Was a ranking done from CP-0 or CP-1 results? Were the results of Native Reach and Common Support separated? Was the same user product sequence balanced? Were different user panels calibrated?

Was the live product result presented as a fundamental model superiority? Were unknown areas considered controlled? Is there a public comparison card available? Is the accountable owner of the system registry known?

53. OBJECTIONS AND RESPONSES

Objection 1 — “If the user does not know which model is running, can this work still be done?”

It can be done. The main GEO-1000 object is the user's product. If the model is unknown, the result is reported as: “This behaviour was observed on this product and user surface.” No reference is made to a specific model.

Objection 2 — “Why do we differentiate the products of the same provider so much?”

Because the user:

• different plan,
• model,
• retrieval,
• memory,
• tool

It may see different answers under different conditions. The provider name does not fully define the user experience.

Objection 3 — “Isn't comparing different providers like comparing apples and oranges anyway?”

If the comparison question is set up correctly, it is not. A real user might ask: “Which product informs me more correctly?” The entire product chain is part of this question. If the superiority of the base model is to be claimed, a more controlled experiment is needed.

Objection 4 — “Isn't it natural for a web-using product to have an advantage?”

It is natural. In a natural product comparison, the advantage is the product's real feature. In model comparison, the retrieval effect should also be separated.

Objection 5 — “Isn't it unfair not to give a citation score to a product that doesn't show citations?”

The attribution feature is the function that the product offers to the user. If the feature is missing, the attribution result can be displayed as:

RECORDED AS NOT AVAILABLE

Material accuracy is also measured separately.

Objection 6 — “If memory is part of the natural user experience, why do we turn it off?”

It can remain on in the natural panel. It can be turned off in the controlled panel to clear comparisons between products. The two panels answer different questions.

Objection 7 — “Isn't opening a new chat enough for a clean session?”

Not always. Persistent memory, special instructions, or account settings may continue. These areas should also be saved.

Objection 8 — “What can we do if the provider did not explain the product change?”

Observed:

• interface,
• feature,
• behaviour,
• Attribution

We can save your changes. If the cause is unknown, indefinite change status such as UC-5 is used.

Objection 9 — "If products are constantly changing, won't any results be valid?"

The result stated:

• date,
• wave,
• Product Sample

It is valid for. Variability does not make measurement meaningless. It prevents untimely judgement.

Objection 10 — "Isn't the same prompt and the same day comparable enough?"

The plan, surface, memory, web, country, and user population may differ. The same prompt may be required. It is not enough on its own.

Objection 11 — 'Wouldn't the CP level be too technical for the public?'

A short label can be used:

• Descriptive
• Partially matched
• Matched
• Controlled
• Replicated

One should know how strong the public ranking is.

Objection 12 — “Why can't we directly rank based on Native Reach scores?”

Because each product may have been measured in different countries and user populations. The score difference may partly arise from which population the product reached. Common Support provides a fairer comparison.

Objection 13 — "Isn't Common Support a real-world result?"

It is the result of the actual shared population. However, it does not show access of the products outside the shared universe. It should be read together with Native Reach.

Objection 14 — “Is live product inspection more important, or frozen model inspection?”

The questions are different. Live product measures the truth of the user. Frozen model provides scientific control and reproducibility. NOMOS does not use them interchangeably.

COMMON RULE OF CHAPTER 57

A user:

• “ChatGPT said so.”
• “Gemini thought so.”
• “Claude recommended us.”
• “Some cited us.”

can be said. These expressions are understandable in daily conversation. They are not sufficient in an audit record. Because under these names:

• different products,
• different plans,
• different models,
• different user interfaces,
• different web and retrieval systems,
• different memory conditions,
• different countries,
• different times

can be found. Different people can see different AI experiences under the same name. Comparable user experiences can be established under different names. A model label seems to have produced the entire answer. But the reality reaching the user:

• source selection,
• security,
• personalisation,
• interface,
• tools

The result may be shaped by security controls, personalisation, the interface and available tools. NOMOS therefore asks more than ‘Which model?’ It asks: Which product? / Which surface? / Which plan? / Which information mode? / Which session? / Which user? / Which country? / Which time? / Which version? Two results may come from the same prompt without coming from the same experiment. Two results may come from different providers yet remain fairly comparable under the same user conditions. A product is a live system: its sources, interface and model can change. Change does not make a result worthless; it binds the result to a particular time and product state. NOMOS's ninth law of measurement is therefore:

An AI provider is not an AI product.

The tenth law is:

The model is only one part of the representation chain that reaches the user.

The eleventh law is as follows:

The same name guarantees comparability; a different name does not guarantee non-comparability.

The twelfth law is as follows:

The same prompt is not the same experiment.

The thirteenth law is as follows:

The outcome of the living product is valid only for the product sample and time at which it is measured.

The fourteenth law is as follows:

The power of comparison comes not from the magnitude of ranking, but from the clarity of the matched conditions.

Order of Section 9 of NOMOS

Do not give me only the provider's name. / Show which product was used, on which surface and with which plan.

Do not make the model label you see on the screen the exact identity of the entire back-end system.

It was used because there was a web feature; do not assume it supported the claim because there was a citation.

Do not count tool access as tool use, a new chat as a clean account, or unknown memory status as memory off.

Do not put a free user and a paid user, a mobile product and a web product, or an enterprise workspace and general chat in the same cell.

Do not automatically consider results from the same provider as a fair comparison.

Do not avoid comparing different providers just because they are different. / First, match the conditions. / Then, explain which question you answered.

Do not present a live product as a frozen model, an API as a consumer experience, or a product result as base model superiority.

If the system changed in the middle of the wave, do not merge the old and new responses as a single product.

Do not ignore the break in behaviour in front of you just because the provider did not explain the change.

Do not assume the unknown is clean, equal, or fixed.

First, register the system in the record. / Then lock the configuration. / Then monitor the changes. / Then write the object of comparison. / Then show matching and non-matching conditions. / And only then say which product gave a stronger result under which condition.

The Chapter's Closing Sentence

The “AI” measured in GEO-1000 is not a single name or model; it is a versioned representation system reaching a specific user at a specific moment within a specific product chain.

Normative Core

Every GEO-1000 observation MUST be linked to a versioned AI System Register record describing the measured product instance, including as applicable: - provider, - product, - user surface, - plan, - displayed or verified model identity, - model-identity confidence, - retrieval and web configuration, - citation capability and use, - tool access and use, - session state, - memory, - custom instructions, - prior context, - personalisation, - country, - locale, - language, - deployment state, - measurement time, - and known or observed system changes. Provider name, product name, displayed model label, base model, retrieval system, and user-facing product MUST NOT be treated as interchangeable identities. Feature availability MUST remain distinct from feature use. Live-product results MUST NOT be represented as frozen-model results, and frozen-model results MUST NOT be represented as full consumer-product experience. Comparisons MUST define their object and receive a disclosed comparability level based on matched entity, prompt, population, language, country, time, session, plan, interface, retrieval, tools, reference records, and adjudication rules. The same provider or displayed model MUST NOT create an automatic assumption of comparability. Different providers MAY be compared when user-facing conditions are sufficiently aligned and the resulting claim is limited to the measured product behaviour. Material product changes during a measurement wave MUST create a separate system state, sub-wave, modelled change factor, or mixed-state warning. Unknown configuration fields MUST remain UNKNOWN and MUST NOT be imputed as controlled or equivalent. Every system-registry and comparability decision MUST be versioned and attributable to an accountable human or organisation.