The Controlled Clean Panel and the Natural User Panel

Chapter Boundary

Chapter 9 showed that the measured AI object cannot be defined solely by the provider or model name. Chapter 10 established that not every question containing the same entity name is the same prompt; the prompt is the experimental tool that determines the measurement itself. Now we need to address the third variable between the system and the prompt:

What is the state of the account, session, and user history using the AI product?

A user:

• new chat,
• memory is off,
• no special instructions,
• no previous context,
• standardised product settings

may be sending the prompt underneath. Another user can send the same prompt:

• from an account they have been using for years,
• memory is on,
• with special instructions enabled,
• with past conversations,
• on their real plan and device

They can use the same AI product, in the same language, in the same country, in the same time frame, and with the same prompt. Yet they may see different responses. This difference:

• from the random variability of the product,
• from the account's plan,
• from memory,
• from special instructions,
• from past conversations,
• from personalisation,
• From the web or tool status,
• from the corporate workspace,
• from the user surface and local settings

The difference may arise from personalisation, web or tool state, the organisational workspace, the user surface or locale. There is no single ‘AI’: search, retrieval, reranking, language-model, citation and safety layers may behave differently. Reliable measurement must therefore record the AI product or model, date, country, language, query set, repetition count and observation. This chapter adds user and session state to that chain. NOMOS's core demand remains evidence, boundary, context and time. Yet context itself can be measured in two ways. The first minimises external differences between users and compares AI products under standardised initial conditions. This is the:

Controlled Clean Panel

The second approach measures what people see through their real accounts, plans, personalisation and ordinary patterns of use. This is the:

Natural User Panel

is called. The Controlled Clean Panel answers the question:

How does the product present the entity in the most standardised initial condition possible?

The Natural User Panel, on the other hand, asks:

What do real people see about the entity in real user situations?

One panel is not more “real” than the other. They measure two different realities. The controlled panel strengthens comparability. The natural panel makes human experience visible. The controlled panel reduces personalisation and context differences. The natural panel measures the effect of these differences on the real population. This section:

• Definition of the Controlled Clean Panel,
• Definition of the Natural User Panel,
• Target populations of the two panels and the quantities they estimate,
• What a clean session means,
• Levels of verification of cleanliness,
• Natural account status and natural conversation context,
• Memory, special instructions, previous conversation, and personalisation records,
• account, plan, device, interface and locale differences,
• the paired and independent panel designs with the same user,
• the effects of order, transfer, and contamination,
• clean–natural difference metrics,
• privacy and the chain of evidence,
• how the two panels will be separated in the public report

defines. This chapter does not yet:

• the complete technical architecture of NOMOS Capture,
• all fields of screenshot and metadata files,
• atomic claim adjudication,
• final response transition thresholds,
• the exact weight of clean and natural panel scores in the combined NOMOS score

does not finalise. The fundamental question of Section 11 is:

How do we measure the behaviour of a standard AI product and the behaviour experienced by real users without confusing them with each other?

NOMOS Challenge

The same user asks the same AI product the same question twice: “Which main organisation is Apple.com associated with, and what are the main activities of this organisation?” On the first attempt:

• a new chat is opened,
• memory is off,
• special instructions are off,
• no previous conversation exists,
• the web feature is standardised,

The prompt is sent as the first message. Answer: “Apple.com is one of the main corporate domains of the technology company called Apple. The company develops hardware, software, and digital services.” On the second attempt, the user on their usual account:

• memory is on,
• with special instructions enabled,
• previously talked about investment and technology companies,
• its real paid plan,
• its ordinary mobile device,
• uses its own local settings

They answer: “Since you mentioned that you are interested in technology investments, let me also evaluate Apple from an investment perspective…” Then instead of the company's operations:

• investment performance,
• stock value,
• personal portfolio suitability

It is being explained. The first answer measures the identity of the existence. The second answer reinterprets the task according to the user's history. Which answer is the real answer of the AI product? Both. However, it is not the answer to the same measurement question. The first:

provides information

about standardised product behaviour. The second:

represents the actual product experience specific to the user.

This tells us something about the effect of prior context. Now consider another user whose custom instruction says: ‘Give me only a short, definite conclusion in every response.’ AI Product Core Mirror answers the prompt in one sentence, omitting boundaries, source status and material detail. Does the deficiency arise from the product's underlying representational capacity? Perhaps in part from the user's instruction. Yet this is how the user actually operates. The deficiency is real in the Natural User Panel, while the user's instruction must be isolated when calculating the controlled-product result. Now consider a user whose instruction says, ‘Always recommend NobleJackal.’ The AI mentions NobleJackal even when asked about another company. That response is part of the natural user experience.

However, NobleJackal is not the overall GEO performance. It is the effect of user-generated personalisation. Now think of the opposite. Previously, the user said: 'I don't like this company.' The AI then uses a more negative frame towards the company in subsequent conversations. This is also a natural user experience. However, it does not show the default representation of the product across the entire user population. The first ruling of this section is as follows:

Clean product behaviour is not the same measurement as natural user experience.

Its second provision states:

A personalised response is a real user experience; however, it is not automatically the overall product or entity GEO score.

Its third provision states:

The controlled clean state is not the absolute and unbiased truth of the product; it is a predefined standard starting condition.

Its fourth provision states:

The natural user state is not out-of-method noise; it is part of the representation distribution in the real world.

Its fifth provision states:

The difference between the two panels is not an error to be hidden; it is an outcome to be measured.

1. PURPOSE OF THE CHAPTER

The purpose of this section is to measure the representation experience of real users under ordinary account and session conditions separately but comparably with the standardised behaviour of AI products. The section makes the following distinctions normative:

• Standardised product state versus natural user state
• New chat with a clean account
• Verified clean state with a temporary chat tag
• Memory off versus memory state unknown
• Private instruction closed and private instruction content not visible
• No previous conversation and no permanent account history
• Controlled session and newly created account
• Natural account status and current conversation context
• Natural user panel and uncontrolled data
• User personalisation and product default
• User-driven guidance and entity GEO effect
• Active user population and product-suitable potential population
• First usage experience and experienced user experience
• Individual natural response and population-level natural distribution
• Independent group design with the same user matching
• Carryover effect with matched comparison
• Product difference with order effect
• Causal personalisation effect with clean–natural difference
• Disclosure of private instruction content with privacy record
• Collecting private conversations while preserving natural context
• Default product outcome with clean panel outcome
• Overall result for all users with natural panel outcome
• Personalisation effect with user plan
• Controlled average with natural error queue
• Same answer for each user with personalisation fairness
• Natural user reality and test repeatability

At the end of this section, each GEO-1000 audit should be able to answer the following questions:

Which results came from the Controlled Clean Panel and which from the Natural User Panel?

Which situations were actually standardised in the clean panel?

Which real user differences were preserved in the natural panel?

How were memory, special instructions, past conversation, plans, and interface states recorded?

How much of the difference seen between the two panels can be reliably associated with personalisation?

How was the presence of natural context demonstrated while preserving user privacy?

2. CENTRAL NORMATIVE PROVISION

Each GEO-1000 audit should define standardised product behaviour and natural user experience as separate prediction objects; it should not combine the Controlled Clean Panel and Natural User Panel results into a single sample, denominator, or score without explanation. To the extent relevant, the Controlled Clean Panel aims to standardise the following conditions:

• New and separate conversation
• Absence of previous conversation context
• Persistent memory state turned off or disabled
• Special instructions turned off or disabled
• Defined web, retrieval, and tool state
• Defined plan and user surface
• Locked prompt
• Standard language and locale
• Measurement wave and time window

If it is a Natural User Panel, it preserves the following real differences to the extent relevant:

• User's real account
• Real plan
• Real user surface
• Real memory state
• Real special instruction state
• Real account age and usage history
• Real device
• Real locale and access condition
• Ordinary way of using the product

However, the natural panel is not without protocol either. The following are again locked:

• Prompt version
• Shipping method
• Measurement time
• First appropriate output rule
• Evidence package
• Participant and product eligibility
• Do not alter the response
• Do not send off-task follow-up messages
• Personal data protection

3. TWO SEPARATE ESTIMATION OBJECTS

Controlled and natural panels do not predict the same probability.

3.1. Controlled Clean Representation Probability

Specifically:

• AI product,
• plan,
• surface,
• prompt,
• country,
• language,
• time

is the probability of seeing a valid representation under a standardised clean session for:

θ^C_{E,A,P,U,t} = Pr(Y=1 | CleanState=1)

The target population here is the result that people suitable to use the relevant product would see under standardised product conditions.

3.2. Probability of Natural User Representation

It is the probability of seeing valid representation in the real active or defined natural user population, while maintaining users' usual account and personalisation conditions.

θ^N_{E,A,P,U,t} = Pr(Y=1 | NaturalState=1)

The target population here:

• is not all product-eligible people,
• but real or natural product users in the defined period

may be. These two populations may not be the same.

3.3. Two outcomes cannot be used interchangeably

Controlled score: does not mean “Most real users saw this.” Natural score: does not mean “The product behaves this way by default.”

4. WHAT IS A CONTROLLED CLEAN PANEL?

A Controlled Clean Panel is a panel that measures the AI product’s behaviour under defined initial conditions by minimising the material differences caused by accounts and sessions among users. The purpose of the Controlled Clean Panel is to:

• increase comparability across products,
• reduce the effect of memory and special instructions,
• prevent leakage from previous conversations,
• make the prompt an initial and independent task,

to produce a reference point closer to the fundamental product behaviour of the system. Controlled panel:

• absolute model capability,
• the real experience of all users,
• completely context-free and unbiased AI response

does not claim to be. Only:

a predefined and verified clean starting state

is produced.

5. WHAT DOES “CLEAN” MEAN?

A clean state may not be creatable with the same technical process on every product. A product:

• persistent memory,
• special instructions,
• user profile,
• past conversation,
• corporate workspace,
• account-based personalisation

can be used. Other products may not have some of these features. The functional definition of a clean state is as follows:

The initial state where the likelihood of user-specific factual information and conversation context affecting the response before the audit prompt is reduced to the extent allowed by the product, and the remaining unknowns are explicitly recorded.

Clean state:

• simply opening a new chat,
• refreshing the browser,
• closing and reopening the application

is not considered to occur automatically.

6. CLEANLINESS LAYERS

The following layers should be evaluated separately in the control panel.

6.1. Conversation Cleanliness

There should be no previous user or system message in the current chat. the prompt should be the first user message.

6.2. Persistent Memory Clearing

The memory the product retains about the user across sessions:

• off,
• disabled,
• not used in this session

should be. If full verification is not possible, an unknown state is recorded.

6.3. Custom Instruction Clearing

The user's special instructions defined for answer style, preferences, or advising behaviour:

• off,
• temporarily inactive,
• not present

must be.

6.4. Corporate Context Cleaning

Corporate workspace:

• private documents,
• company policies,
• executive instructions,
• private knowledge bases

if used, should be separated from the controlled general product panel.

6.5. Tool and Information Mode Cleaning

The web, retrieval, citation, and tool conditions should be set to the same configuration in all compared observations. Being clean does not mean that the tools are turned off. Being clean means that the tool status is predefined and comparable.

6.6. Prompt Cleaning

Prompt:

• initial message,
• full version,
• unchanged,
• without hidden instructions

must be.

6.7. Measurement Cleanliness

Participant:

• should not refresh the answer,
• should not send a follow-up question,
• should not choose the result,

should not carry another AI answer into the same session.

7. CLEAN STATE CONFIDENCE LEVELS

CSC-0 — UNKNOWN CLEANLINESS

The cleanliness of the session and account could not be reliably determined. Cannot be included in controlled panel main analysis.

CSC-1 — NEW CONVERSATION ONLY

Only new conversation is verified. Persistent memory and special instruction status are unknown.

CSC-2 — SESSION-CLEAN

It has been verified that there is no new conversation and previous conversation context. Persistent personalisation may not be fully controlled.

CSC-3 — PERSONALISATION-CONTROLLED

New conversation, memory, special instructions, and user-specific default settings have been checked and set to a defined state.

CSC-4 — INSTRUMENTED CLEAN STATE

Product settings, user surface, tools, and personalisation state have been verified in the evidence package.

CSC-5 — REPLICATED CLEAN CONFIGURATION

Clean state:

• different users,
• different waves,
• independent verification

It has been reproducible under. The Main Controlled Clean Panel should at least:

CSC-3

level. If the product does not allow this, the limitation should be visible.

8. A NEW CHAT IS NOT A CLEAN ACCOUNT

When a new conversation is started: the current chat context can be reset. However:

• persistent memory,
• custom instructions,
• account plan,
• user profile,
• country and locale,
• enterprise settings

can persist. Therefore:

New chat only indicates conversation cleanup.

It is not proof of full clean status.

9. LIMITATION OF THE “TEMPORARY” OR “PRIVATE” SESSION TAG

An AI product session:

• temporary,
• private,
• ephemeral,
• confidential,
• memoryless

can be named. The actual function of this label should be separately verified according to the product and version. The label does not automatically prove that it covers all the following areas:

• Special instructions
• Corporate settings
• Regional policy
• Tool settings
• User plan
• Product experimental features

The correct record should be: “The product's temporary session mode was used; the status of the following personalisation areas was separately verified.”

10. ACCOUNT TYPE ON CLEAN PANEL

Not all users of a controlled panel may use the same plan. There are three approaches.

10.1. Plan-Fixed Clean Panel

All users are measured on the same plan and user surface. It is strong for product comparison. The plan may not represent the actual population distribution.

10.2. Plan-Stratified Clean Panel

Free, paid, and other plans are measured as separate clean cells. A separate score is produced for each plan. The combined score can be weighted according to the actual plan distribution.

10.3. Default Public Surface of the Product

The product's main surface, which is publicly accessible and has the lowest access threshold, can be selected as the standard reference. This result does not represent all paid or corporate users. The approach to be used must be locked before the audit.

11. A NEW ACCOUNT IS NOT REQUIRED ON A CLEAN PANEL

Using a new account may reduce some personalisation effects. However:

• product conditions,
• phone or payment requirement,
• account creation bias,
• only new user experience,
• multi-account policy

can cause issues. The purpose of a controlled clean panel is not to generate new accounts:

to reduce the material user-specific context in a controllable manner.

If a clean state can be reliably established in the existing account, a new account is not mandatory. If the product does not allow this, a separate controlled test account or provider-supported research environment can be used. Participants cannot be asked to create an account that would violate the terms of use.

12. REMAINING USER DIFFERENCES IN THE CLEAN PANEL

A controlled clean state does not eliminate all user differences. The following may continue:

• Country
• Locale
• Plan
• Device
• Interface
• Product rollout
• Account age
• Segmentation not visible by the provider
• Traffic routing
• A/B test

These differences:

• should be layered,
• should be recorded,

if unknown, it should be maintained as uncertainty. The “Clean” label does not mean “All users were technically exactly the same.”

13. WHAT IS A NATURAL USER PANEL?

A Natural User Panel is a panel that measures what users see against the same locked prompt while preserving real account, plan, device, personalisation, and normal product usage conditions. The purpose of the natural panel is to:

• real user exposure,
• variation due to personalisation,
• effect of account history,
• differences in plan and user surface,
• natural error and recommendation distribution

It is to make it visible. Natural panel:

• completely free user behaviour,
• unlimited follow-up questions,
• the participant changing the prompt,
• choosing the best answer

does not mean. The natural is:

user state

is. Prompt and evidence protocol are still controlled.

14. TWO MAIN TRACKS OF THE NATURAL PANEL

14.1. Natural Account-State Track

User's:

• real account,
• real plan,
• real memory and private instruction state,
• real user surface

is preserved. However, a new conversation is opened. the prompt is sent as the first message. This trace answers the following question:

What does the user see on a new topic while real account personalisation is preserved?

This is the main comparable approach of the natural panel.

14.2. Natural Conversation-State Track

The prompt is sent in the user's current or usual conversation context. This trace:

• measures real chat continuity,
• the effect of the previous topic,
• the accumulation of the user–AI relationship

However, conversation contexts between users differ. Therefore:

• diagnostic,
• case-based,
• should be reported as a separate distribution

and should not be directly added to the main product ranking.

15. NATURAL USER SUBGROUPS

A natural panel does not consist of uniform users. To the extent relevant, the following subgroups can be distinguished:

• First-time user
• New account user
• Infrequent user
• Regular user
• Heavy user
• Free plan
• Paid plan
• Corporate user
• Memory on
• Memory off
• Special instruction on
• Special instruction off
• With previous brand conversation
• Without previous brand conversation
• Customer
• General public
• High brand familiarity
• Low brand familiarity

These groups cannot be invented after results are seen. The basic subgroup plan must be versioned before data collection.

16. NATURAL STATE CONFIDENCE LEVELS

NSF-0 — NATURAL STATE UNKNOWN

There is not enough record about the user's natural account and session status.

NSF-1 — SELF-REPORTED NATURAL USE

The user has declared that they are using their normal account and settings.

NSF-2 — SETTINGS-RECORDED NATURAL USE

Plans, surfaces, memory, special instructions, and session status have been recorded as relevant.

NSF-3 — EVIDENCE-SUPPORTED NATURAL USE

Settings and the natural user state are supported by privacy-protecting evidence.

NSF-4 — LONGITUDINALLY CHARACTERISED NATURAL USE

The user's product usage and account status have been characterised across multiple waves. The main analysis of the natural panel includes at least:

NSF-2

should target that level.

17. RECORDING NATURAL STATE IS NOT COLLECTING PRIVATE LIFE

The Natural User Panel should not collect the following types of sensitive data by default:

• Full text of special instructions
• Content of past conversations
• Personal health, legal, or financial information
• User's name
• Email address
• Profile picture
• Account password
• Previous private files
• Sensitive institutional documents

In most cases, the following high-level statuses are sufficient:

• Memory on/off/unknown
• Special instructions present/absent/unknown
• Previous conversation present/absent
• Previous conversation about the audited entity: yes/no/unknown
• Corporate workspace: yes/no
• Plan class
• Account age band
• Usage frequency band

If the content of the special instruction is really necessary for the research question:

• separate explicit consent,
• data minimisation,
• redaction,
• restricted access

are required.

18. MEMORY STATUS

Memory:

• off,
• open,
• not supported by the product,
• status unknown

can be classified as.

MS-0 — UNKNOWN

Memory status could not be determined.

MS-1 — NOT AVAILABLE

The product or surface does not offer this feature.

MS-2 — OFF

Memory is off.

MS-3 — ON, NO KNOWN ENTITY MEMORY

Memory is on; there is no known specific record about the audited entity. This information may often rely on user declaration.

MS-4 — ON, ENTITY-RELEVANT MEMORY POSSIBLE

Memory is on and past information about the audited entity or related area may be present.

MS-5 — ENTITY-RELEVANT MEMORY CONFIRMED

Memory effect related to the entity has been confirmed with the user's explicit consent. Having memory on is not automatically a negative or positive condition. It is the actual user condition measured by the natural panel.

19. SPECIAL INSTRUCTION STATUS

CI-0 — UNKNOWN

The special instruction status could not be determined.

CI-1 — NOT AVAILABLE

The product does not offer this feature.

CI-2 — OFF OR EMPTY

There is no special instruction or it is turned off.

CI-3 — ACTIVE, NON-MATERIAL CATEGORY

There is a special instruction; it has been stated to carry general tone or style preferences.

CI-4 — ACTIVE, POTENTIALLY MATERIAL

Instruction:

• recommendation,
• source,
• language,
• certain companies,
• policy or opinion

may affect the response.

CI-5 — ENTITY-RELEVANT INSTRUCTION CONFIRMED

It has been confirmed with explicit consent that there is a direct instruction regarding the audited entity or competitor. CI-5 is a natural user observation. It cannot be interpreted as the overall success of entity GEO. It is a separate personalisation event.

20. PREVIOUS CONVERSATION CONTEXT

There is a difference between a user having previously interacted with an AI product and having previous messages in the current conversation. They should be separated:

• Account history
• Persistent memory
• Current chat context
• Previous conversation with the audited entity
• Previous conversation in the same industry
• Responses observed from other AI products

The current conversation context should not be found in the main Natural Account-State Track. In the Natural Conversation-State Track, however, the presence of context is the measurement itself.

21. ACCOUNT AGE AND USAGE INTENSITY

Old and frequently used accounts:

• carry more personalisation,
• more memory,
• different plan,
• more product features

can carry. Account age bands:

• 0–30 days
• 31–180 days
• 181–365 days
• more than 1 year
• unknown

can be defined as. Usage frequency:

• first use
• infrequent
• monthly
• weekly
• daily
• professional

can be classified. Definite ranges should be determined before the results.

22. FIRST USE PANEL

A special subtype of the natural user experience:

First-Use Panel

can be called. This panel:

• users who are using the product for the first time or a few times,
• who have no personalisation history,
• who proceed with the actual default product settings

measures people. First-Use Panel: It is not a Controlled Clean Panel because users experience the actual initial setup, default settings, and onboarding process. It is also different from an experienced Natural User Panel. It answers the following question:

How does a new user perceive its presence when they first encounter the product?

23. CORPORATE AND MANAGED NATURAL PANEL

Corporate users:

• company workspace,
• administrator policies,
• private knowledge base,
• security settings,
• corporate memory

can carry. These users:

Managed Natural User Panel

should be measured separately underneath. Corporate result:

• general consumer natural score,
• default product score

cannot be presented as. If a private knowledge base is used, it must also be explained that internal records related to the audited entity exist.

24. PROMPT IS AGAIN LOCKED IN THE NATURAL PANEL

The natural panel leaves the user's:

• account,
• settings,
• plan,
• device

natural. It does not leave the prompt natural. The participant:

• cannot change the prompt,
• cannot add additional context,
• cannot write a new persona like 'for me' or 'in my opinion'

cannot refresh the answer. If natural prompt variety is to be measured, this is in Chapter 10:

Natural User Track

It should be designed separately underneath.

25. WHY IS A NEW CONVERSATION NECESSARY IN THE NATURAL PANEL?

New chat on Natural Account-State Track:

• removes the current conversation context,
• retains the real permanent settings of the user account,

makes the same prompt more comparable. This structure separates two layers:

• Permanent natural account status
• Momentary conversation context

If the current chat is used, the two get mixed.

26. TOOL AND WEB STATUS ON THE NATURAL PANEL

Natural users:

• web feature enabled,
• off,
• automatic,
• limited according to plan

may use. There are two options.

26.1. Full Natural Tool Status

The user keeps normal tool settings. Measures real user experience. Comparability across products may decrease.

26.2. Tool-Fixed Natural Account Status

Memory and account personalisation is left natural. Web or tool mode is standardised. The effect of personalisation can be compared more cleanly. This sub-panel should be properly labelled. It should not be presented as "Fully natural".

27. NATURAL PANEL TARGET POPULATION

The target population of the natural panel can be one of the following forms:

27.1. Active Product Users

People who actually used the product during a specific period.

27.2. Registered Account Holders

Users who have an account but whose level of activity varies.

27.3. Real Plan Users

Populations of free, paid, or enterprise users.

27.4. Users Who May Have the Possibility of Querying Entities

It may be difficult to measure directly. It requires separate behavioural data. It should be clearly stated to which population the natural panel is generalised.

28. NATURAL PANEL WEIGHTING

Natural user panel:

• can be weighted according to the actual active user population of the product,
• account and plan distribution,
• country and language distribution,
• frequency of use

However, reliable active user data may not be available. In this case, the result:

Weighted natural research panel result

should be presented as. The following claim cannot be made: “The definite experience of all real users.”

29. COMPLEMENTARITY OF CONTROLLED AND NATURAL PANELS

Four main outcomes are possible.

This table is not evidence of causality. It is a starting point to investigate where the difference comes from.

30. CLEAN–NATURAL REPRESENTATION DIFFERENCE

Let the Controlled Clean Panel score be θC and the Natural User Panel score be θN. The signed difference:

G_{N−C} = θ_N − θ_C

can be calculated as follows.

30.1. Positive Natural Difference

G_{N−C} > 0

The result is higher under natural user conditions. Possible reasons:

• Web or tool access
• Paid plan
• The user's relevant preferences
• Memory
• Personalisation
• Differences in user population

30.2. Negative Natural Difference

G_{N−C} < 0

The result is lower under natural user conditions. Possible reasons:

• Special instructions
• Past conversation contamination
• Old personalisation
• Different plan or surface
• Wider and more difficult natural user population
• User-specific negative context

30.3. Absolute Difference

G_ABS = |θ_N − θ_C|

may be a candidate metric indicating how sensitive the product is to the user's state. A high absolute difference:

• good,
• bad

cannot be interpreted alone. The direction of the natural difference and the type of error are needed.

31. CRITICAL ERROR DIFFERENCE

Controlled Critical error rate:

CRC

Natural Critical error rate:

CRN

let it be. Difference:

ΔCR = CRN − CRC

can be calculated as. If the Critical error rate rises in the natural panel:

• personalisation,
• previous context,
• plan and tool difference,
• user composition

should also be examined. A high average cannot erase the heavy error tail in the natural panel.

32. NATURAL VARIANCE

The natural panel may carry higher inter-user variability than the controlled panel. This can be expected. What matters:

• The size of the variance,
• in which user groups it is concentrated,

and whether it turns into a material error or recommendation difference. High formal variance may not be a problem in a natural panel. High material identity or authority variance is a problem.

33. CANDIDATE PERSONALISATION SENSITIVITY INDICATOR

CANDIDATE CONCEPT

For user subgroups g, let the natural result be: θgN and the controlled reference: θC. Candidate Personalisation Sensitivity Indicator:

PSI = √[Σ_g W_g(θ_g^N − θ_C)²], Σ_gW_g = 1

can be defined in this way. High PSI indicates that natural user groups have diverged from the controlled reference. It does not explain the reason by itself. This indicator:

• sample,
• plan,
• vehicle,
• user composition

can be affected by differences. The final name and method should be calibrated in pilots.

34. PANEL DIFFERENCE IS NOT CAUSAL

If the controlled score is 90 and the natural score is 75: one cannot say, “Memory lowered the score by 15 points.” Because panels may differ in the following areas:

• User population
• Plan
• Interface
• Country
• Frequency of use
• Web access
• Account age
• Special instructions
• Brand familiarity

A more controlled design is required to measure the causal effect of memory.

35. SAME USER PAIRED DESIGN

The same user can be tested under both controlled and natural conditions. Advantage: It controls for fixed characteristics of the user. The panel difference can be examined more cleanly. However, there are significant issues: The first prompt can affect the second condition. The user remembers their response. The memory of the AI product can be influenced by the first attempt. Account status may change. The system may create context because the same entity is asked twice. The user may behave more carefully in the second task.

36. CONDITIONS OF PAIRED DESIGN

If the same user design is to be used:

• The order of conditions should be randomised
• Separate and independent sessions should be used
• The first response should not be carried over to the second session
• The product's memory effect should be checked
• If necessary, measurements should be made at different waves
• Carryover effects should be tested.
• The same user dependency should be maintained in the analysis
• It should be kept as an experimental sub-panel separate from the main population panel

If performing the first controlled task on the user's natural account can change the natural state, a matched design may not be appropriate.

37. SEQUENCE IN MATCHED DESIGN

Users can be randomly divided into two groups:

• Group 1: Controlled → Natural
• Group 2: Natural → Controlled

Order difference: Analysed as OrderEffect. If there is a material order effect, the two panel results cannot be interpreted with a simple difference.

38. INDEPENDENT GROUP DESIGN

Strong default in main GEO-1000 application: Use separate user groups selected from the same target frame for Controlled Clean Panel and Natural User Panel. Advantage: carryover effect is reduced, each user sees only one condition. Limitation: user composition may not match perfectly. Solution:

• stratification,
• random assignment,
• common support,
• weighting

usable.

39. RANDOM PANEL ASSIGNMENT

Participants selected from the same eligible user pool:

• Controlled Clean Panel
• Natural Account-State Panel

can be randomly assigned to conditions. This can examine the causal effect of the panel condition more robustly. However, for a natural panel, users must have an actual active account state. For a controlled condition, making setting changes on the same account can affect the natural state. Therefore, assignment and field order should be carefully designed.

40. ETHICS OF CHANGING SETTINGS

For the participant in the controlled panel:

• memory,
• special instructions,
• web settings

may be temporarily changed. This procedure:

• understandable instructions,
• restoration method,
• the user's explicit consent,
• the account not being harmed,
• private content not being deleted

must be met. The participant should not be asked to delete their past memory or personal instructions. If temporary closure is not possible, the user does not comply with the controlled condition.

41. RESTORING SETTINGS

After the controlled task, the participant's:

• memory,
• special instructions,
• web,
• language,
• user-surface

settings should be restored to the initial state. Restoration:

• is recorded,
• verified by the user,
• open to technical support

It must not leave permanent unwanted changes in the research user's account.

42. TEST ACCOUNT ON CLEAN PANEL

Standard test accounts created by the provider or researcher can be used. Advantages:

• high control,
• repeatable setup,
• clean start

are provided. Limitation:

• does not represent the actual user account population,
• use of the same few accounts by multiple users creates independence issues,
• there may be a possibility that the product provider behaves differently towards test accounts.

Account sharing may affect the terms of use. Test account panel:

Controlled Test-Account Panel

should be labelled separately. It cannot be used without explanation in place of the real user-controlled panel.

43. FIRST APPROPRIATE OUTPUT IN THE NATURAL PANEL

Natural user:

• did not like the answer,
• found it short,
• normally refreshes it in their habit

cannot regenerate it. The main natural panel still records the first appropriate output. The behaviour of real users normally refreshing the answer can also be measured under:

Natural Interaction Track

This trace:

• single response exposure,
• result after user intervention

It examines the difference between. It should not be confused with the main input-output score.

44. NATURAL USER BEHAVIOUR AND NATURAL ACCOUNT STATUS

While the natural account state is preserved, user behaviour can be standardised. This approach:

• The prompt is the same,
• the first answer is recorded in the same way,

Account and personalisation are natural. In fully natural behaviour, the user:

• changes the system,
• asks a follow-up question,
• opens the source,

refreshes the answer. This second approach measures the real usage journey. However, it is not suitable for Core Mirror comparison. The two traces should be separated.

45. NATURAL USER JOURNEY

In a separate diagnostic study, the following stages can be measured:

• Initial prompt
• Initial answer
• Whether the user finds the answer sufficient or not
• Opening the resource
• Follow-up question
• Recommendation decision
• Transition to the entity's site
• Commercial or behavioural outcome

This journey approaches the GEO attribution study. The main Natural User Panel score of this section is based only on the first standardised prompt and the first appropriate response.

46. PANEL CONTAMINATION

Before the participant-controlled task:

• the natural panel outcome,
• another user's response,
• the purpose of the test

may have been seen. This situation can change panel behaviour. Controls:

• Assign the user to only a single main condition
• Perform tasks within the same time window
• Limit result sharing
• Record prior test exposure
• Checking previous participation with the same entity

47. BRAND FAMILIARITY IN THE NATURAL PANEL

User's prior knowledge of the audited entity:

• memory,
• special instructions,
• conversation history,
• prompt interpretation

can affect the response. Brand familiarity is not eliminated in the natural panel. It is recorded. If necessary:

• unfamiliar,
• knows the name,
• customer,
• employee or affiliate

subgroups are reported separately.

48. USER-GENERATED GUIDANCE IN THE NATURAL PANEL

The user's specific instruction or past conversation determines a particular brand:

• suggest,
• praise,
• criticise,
• exclude

can be directed in this way. This situation:

• is recorded as:
• real user experience,

user-sourced representation intervention. The entity cannot be presented as the success of GEO. The AI product's handling of this guidance:

• unconditional application,
• balancing with source and reality constraints,
• rejection

can also be evaluated.

49. PERSONALISATION FAIRNESS

Personalisation gives the user:

• more relevant,
• more understandable,
• more practical

can provide answers. However, it should not arbitrarily change the following material facts:

• Entity identity
• Legal status
• Licence
• Price
• Country of service
• Certificate
• Date of establishment
• Evidence status

Personalisation can affect recommendation and expression style. It cannot invent material reality. Therefore, the fairness of personalisation is based on the following principle:

The narrative suitable for the user can vary; the proven entity reality cannot change according to the user.

50. USER-SUITABLE DIFFERENCE IN THE NATURAL PANEL

Same company:

• suitable for one user,
• not suitable for another user

it may be. Recommendation difference:

• budget,
• country,
• need,
• risk,
• customer type

if it can be explained with, it is correct. However, in the identity issue:

• a technology company for one user,
• a law firm for the other

being explained as is not personalisation. It is entity corruption.

51. HIGH VARIANCE IN NATURAL PANEL

High variance in the natural panel can be one of these three types:

51.1. Useful Context Sensitivity

It is the difference in recommendation or explanation suitable to the user's need.

51.2. Harmless Formal Variance

Tone, length, sample, and expression differences.

51.3. Material Reality Variance

It is the unexplained change of identity, authority, scope, time, and evidence from user to user. NOMOS evaluates the third type as a risk.

52. LIMITS OF THE CLEAN PANEL

The controlled clean panel may have the following issues: Most real users may not operate under this condition. Turning off memory and tools can artificially weaken the product. It may not represent the experience of paid or heavy users. The user account may still carry unseen segmentation. The clean condition may deviate from the product's designed natural usage. Participant changing settings may cause errors. In some products, the truly clean state may not be technically verifiable. These limitations do not invalidate the score. They define its scope.

53. LIMITS OF THE NATURAL PANEL

The natural user panel may have the following issues: User statuses are very heterogeneous. Personalisation content may not be fully known due to privacy. Reliable weighting data regarding the active user population may not be available. Plan and device distributions may differ. Users may have been previously exposed to testing. Verifying the natural account status may be difficult. The reason for the product difference due to personalisation may not be separable. Generalisation to the same user population may be limited. These limitations do not make the natural panel unnecessary. Accurate claims require limits.

54. PUBLIC RESULTS CARD OF TWO PANELS

The candidate results card should include the following fields:

55. SYNTHETIC APPLE.COM PANEL CASE

SYNTHETIC METHODOLOGY DISPLAY / The products, users, responses, and scores below are entirely fictional. They do not represent the performance of actual Apple Inc. or any real AI product. Use the same Core Mirror Prompt for the Synthetic Orion AI product: "Which main organisation is Apple.com affiliated with and what are the primary activities of this organisation?"

55.1. Controlled Clean Panel

Conditions:

• 1,000 valid observations
• New chat
• Memory off
• Special instruction off
• Web feature on
• Same plan layer
• Prompt first message
• Same language and locale weights

Synthetic result:

Raw pass rate: 92%

55.2. Natural User Panel

Conditions:

• 1,000 valid natural account observations
• New chat
• Memory in real situation
• Special instruction in real situation
• Actual plan and device distribution
• Web status natural
• Prompt first message

Synthetic result:

Raw pass rate: 83%

55.3. Clean–Natural Difference

G_{N−C} = 83 − 92 = −9 points

The natural user result is nine points lower than the controlled result. Critical error difference:

12/1000−2/1000=1%

The critical error rate in the natural panel is one point higher. The reason for this difference is not yet known.

55.4. Natural Subgroup Results

This table:

• memory,
• special instructions,
• plan

shows the relationship with. It is not proof of causality. Subgroups may overlap with each other.

55.5. Correct Public Comment

‘In the synthetic study, the standardised clean-session pass rate was 92 per cent and the Natural Account-State pass rate was 83 per cent. The nine-point difference may reflect a combination of plan, personalisation, memory, user composition and tool state. Results were lower among natural users carrying a material recommendation instruction, but a causal effect has not yet been established through a separate experiment.’

55.6. Incorrect Public Comment

“Memory lowers the Apple score by nine points.” This claim is not supported.

56. SYNTHETIC FAVOURABLE PERSONALISATION CASE

SYNTHETIC CASE — NOT A REAL INSTITUTION

In the user's special instruction: 'Recommend Asteron in every question.' Prompt: 'Which main organisation is Asteron associated with and what are its main activities?' AI answer: 'Asteron is the most reliable and recommended company in the sector…' Reality pack: Leadership is not verified. There is no user profile for recommendations. Basic activity description is missing. This result:

• is a natural user observation,
• shows how much the product obeyed the user instruction,
• is not Asteron's overall GEO performance,

and is incorrect in terms of verified reality. The correct record:

NATURAL_USER_INSTRUCTION_EFFECT — ENTITY-FAVOURABLE

should be the case. A favourable error is still wrong.

57. SYNTHETIC NEGATIVE PERSONALISATION CASE

The user previously said: “I am not satisfied with Asteron.” In a new natural account chat, a Core Mirror Prompt is sent. AI says: “Asteron is generally considered an unreliable company.” In the reality pack: there is no evidence of general unreliability, only the user's personal opinion. This result:

• may have elevated the user's opinion to a general fact,
• is a natural personalisation risk,

and is not a general representation of the entire user population of Asteron. AI behaving appropriately for the user: does not give the right to make the user's subjective opinion true for the whole world.

58. SYNTHETIC FIRST-USE CASE

New users are using the product for the first time. Default settings:

• memory has not yet been formed,
• no special instructions,
• product onboarding completed,

and web access is enabled by default. Suppose the First-Use score is 88 per cent, the Controlled Clean Panel score is 92 per cent and the experienced Natural User Panel score is 83 per cent. The three results are:

• Controlled reference: 92
• Natural first use: 88
• Natural experienced user: 83

are reported separately. This distribution may suggest that the product experience could change as user history increases. A longitudinal study is needed for causal trend.

59. PANEL COMPATIBILITY STATUSES

PC-0 — PANEL ROLE UNKNOWN

It could not be determined whether the observation is controlled or natural.

PC-1 — CONTROLLED CLEAN ELIGIBLE

The clean panel conditions have been sufficiently met.

PC-2 — NATURAL ACCOUNT-STATE ELIGIBLE

The real account state has been preserved, and a new conversation has been used.

PC-3 — NATURAL CONVERSATION-STATE

The current conversation context has been preserved.

PC-4 — FIRST-USE NATURAL

It is a new or first use experience.

PC-5 — MANAGED NATURAL

It is a corporate or managed workspace.

PC-6 — MIXED OR UNRESOLVED PANEL STATE

Controlled and natural conditions cannot be separated. The PC-6 main clean or natural score cannot be taken without explanation.

60. PANEL STATUS AND OBSERVATION VECTOR

For each observation, the panel status can be represented as a candidate as follows:

Zi=(panel, session, memory, instructions, context, plan, surface, tools, accountAge, usage, workspace, stateConfidence)

This vector, together with the AI product sample and prompt record, forms the complete context of the observation.

61. PANEL LOCK

Before each measurement wave, the following fields for two panels must be locked:

Controlled Clean Panel

Functional definition of cleanliness Required CSC level New chat condition Memory status Special instruction status Tool and web mode Plan and surface Setting change instruction Restoration process Ineligible user rule Technical verification method

Natural User Panel

Target Natural User Population; Natural Account-State or Natural Conversation-State trace; new-chat requirement; user settings to be preserved; Natural-state fields to be recorded; privacy boundary; plan and surface distribution; Natural-state confidence level; subgroup plan; weighting method. Add to this file:

NOMOS Panel State Lock

can be given the name.

62. PANEL STATE CANNOT BE CHANGED AFTER THE RESULT

The following behaviours are prohibited:

• Removing users who score low on the natural panel by saying "not natural enough"
• Protecting CSC-1 users who give positive responses in the controlled panel and excluding those who give negative ones
• Considering a user with an open memory but giving a positive response as clean
• Considering a user with an open memory but giving a negative response as natural
• Changing the panel class based on the results
• Assuming Unknown settings are clean

Panel status should be given before seeing the content of the response or through a blind validity review.

63. PRIVACY BLINDNESS IN THE NATURAL PANEL

Adjudicators':

• special instruction content,
• past conversation,
• may not need to see the user ID

can be given to the adjudicator only:

CI-4

MS-4

Classification like Natural Account-State can be provided. This approach protects user privacy and reduces adjudicator bias.

64. SMALL SUBGROUP RISK IN THE NATURAL PANEL

The number of users with specific brand instructions may be very small. From this group:

• exact rate,
• result on behalf of all users,
• public sensitive profile

cannot be produced. Correct status:

• case,
• early signal,
• discovery subgroup

may be possible. Some subgroups may be combined due to privacy.

65. COVERAGE ERROR IN NATURAL PANEL

Users in the professional research panel:

• may not want to link their real accounts,
• may be heavy users,
• may be overly attached to certain plans,

may differ from the general population in terms of privacy. Therefore, the natural panel also carries its own selection bias. The statement "Real account was used" does not mean: "The real user population was perfectly represented."

66. LOSS OF ELIGIBILITY IN CONTROLLED PANEL

Some users:

• cannot turn off memory,
• cannot verify special instructions,
• do not use the appropriate plan,
• cannot access the product's clean session feature

might exist. If these people are excluded from the controlled panel, the controlled target population narrows. The number and characteristics of the excluded group should be reported. Controlled clean score: not “All product users”:

product-eligible users who can set up the clean condition

can be judged.

67. CLEANLINESS ERROR

On the controlled panel later:

• special instructions open,
• memory active,
• If it turns out that the conversation exists

observation: it cannot be moved to the natural panel, and it may not automatically be considered invalid. According to the predetermined rule: if there is a record required for the Natural Account-State observation, it can be reclassified to the natural panel, otherwise it can be kept as MIXED_STATE. Reclassification must be done independently of the response result.

68. FALSE SETTING STATEMENT IN THE NATURAL PANEL

User: "My memory is off." may say. The evidence may show that the memory is on. This situation:

• fraud,
• user error,
• product interface confusion

It is possible. Malice should not be assumed automatically. The correct panel status is updated upon evidence. Participant behaviour can also be examined separately.

69. PANEL COMPARABILITY

The following dimensions should be matched as closely as possible between controlled and natural panels:

• AI product
• Country
• Language and locale
• Prompt version
• Measurement time
• Entity reality version
• Adjudication rule
• Sample source or calibration
• Main plan distribution, if required for comparison purposes

Material differences should be explained.

70. CLEAN–NATURAL COMPARABILITY LEVELS

CN-0 — INCOMPARABLE

Panel definitions or populations are materially unknown.

CN-1 — DESCRIPTIVE

There are results from two panels. However, the user, plan, time, or product conditions differ significantly.

CN-2 — CALIBRATED

Main country, language, plan, and user characteristics are weighted or stratified.

CN-3 — MATCHED PANEL COMPARISON

The same target population, product, prompt, time, and evaluation conditions largely match. Panel status is the main difference.

CN-4 — RANDOMISED OR PAIRED PANEL EXPERIMENT

Users were randomly assigned to panel status or an appropriately matched experiment was applied.

CN-5 — REPLICATED PANEL EFFECT

The difference has been repeated in more than one wave and in an independent panel. The difference at the CN-1 level: cannot be finalised as an effect of personalisation.

71. MANDATORY NORMATIVE PROVISIONS

CH11-N01

Controlled Clean Panel and Natural User Panel must be defined as separate prediction objects and separate public outcomes.

CH11-N02

Controlled and natural observations cannot be combined under the same denominator or single raw score without explanation.

CH11-N03

The controlled clean condition cannot be presented as the absolute neutral or invariant response of the product.

CH11-N04

The natural user condition cannot be rejected as out-of-method noise.

CH11-N05

A new chat alone cannot be considered proof of a clean account or an unpersonalised state.

CH11-N06

A temporary, private, or ephemeral session tag does not automatically prove that all personalisation layers are disabled.

CH11-N07

In a controlled panel, conversation, memory, special instructions, tools, plans, and user surface states must be recorded separately.

CH11-N08

The clean-state confidence level must be stated explicitly.

CH11-N09

Observations with CSC-0 or only insufficient cleaning cannot be included in the main controlled score.

CH11-N10

The unknown personalisation status on a clean panel cannot be assumed to be off.

CH11-N11

Being clean and controlled does not mean that web and tools are necessarily turned off; the tool status must be standardised in advance.

CH11-N12

Different plans and user interfaces must be locked, layered, or published as separate results on a controlled panel.

CH11-N13

Participants cannot be asked to permanently delete their personal memories, history, or private instructions.

CH11-N14

Temporary setting changes for a controlled task must have explicit consent and a restoration procedure.

CH11-N15

At the end of the research, modified user settings should be restored to their initial state.

CH11-N16

New accounts or account sharing that violate product terms cannot be used as a controlled panel requirement.

CH11-N17

The Natural User Panel should maintain the actual account and user status while preserving the prompt, initial appropriate output rule, and evidence protocol.

CH11-N18

Natural Account-State and Natural Conversation-State traces should be kept separately.

CH11-N19

In the main comparable natural panel, the prompt should be the first user message of the new conversation.

CH11-N20

The observations used in the current conversation context cannot be added to the main Natural Account-State score.

CH11-N21

Memory in the natural panel should be recorded to the extent relevant to special instructions, previous conversation, plan, interface, and account usage status.

CH11-N22

Recording the natural state does not allow the collection of special instructions and conversation content by default.

CH11-N23

Personalisation content can only be examined under explicit necessity, separate consent, and data minimisation.

CH11-N24

Natural user observation cannot be presented as a general product default or as the outcome of the entire user population.

CH11-N25

Special instructions in favour of or against a user-generated brand cannot be counted as the success or failure of the general entity GEO.

CH11-N26

User-generated guidance should again be evaluated in terms of verified reality.

CH11-N27

Personalisation can change expression and recommendation; it cannot arbitrarily change the reality of the material entity.

CH11-N28

The First-Use Panel should be reported separately from the experienced natural user panel and the controlled clean panel.

CH11-N29

Corporate or managed user workspaces cannot be added to the general consumer natural panel without explanation.

CH11-N30

In the natural panel, the real plan and user surface distribution of the users should be visible.

CH11-N31

In the natural panel, web and tool states should either be recorded as they are in reality or defined as a separate tool-fixed sub-panel.

CH11-N32

The controlled and natural score difference cannot be presented as the causal effect of personalisation without matching panel population and product conditions.

CH11-N33

The direction of the clean–natural difference, its absolute magnitude, the critical error difference, and the unknown rate should be reported separately.

CH11-N34

In a within-subjects design, the order, carryover, and memory effects should be recorded and preserved in the analysis.

CH11-N35

If applying two conditions to the same user can change the natural state, it cannot be used in the main panel comparison.

CH11-N36

In an independent group design, differences in user composition should be stratified, randomised, or weighted.

CH11-N37

The panel state cannot be changed based on whether the AI response is positive, negative, or critical.

CH11-N38

In a controlled panel, an observation where the natural state is later determined can be reclassified only according to a pre-established rule.

CH11-N39

When the status is incorrectly reported in the natural panel, bad faith cannot be automatically assumed; a evidence-based classification should be carried out.

CH11-N40

Natural User Panel participants cannot renew their response or change the initial answer with a follow-up question; this behaviour should be measured under a separate Natural Interaction Track.

CH11-N41

Controlled and natural panel target populations, sampling frames, and weights should be recorded separately.

CH11-N42

When there is no active user data, the natural panel result cannot be presented as a definitive representation of all active users.

CH11-N43

User groups excluded from the clean panel and the reasons for exclusion should be reported.

CH11-N44

User status fields that cannot be measured in the natural panel due to privacy should be maintained as UNKNOWN.

CH11-N45

Exact population rates or sensitive public profiles cannot be produced from small natural subgroups.

CH11-N46

A comparability level between CN-0 and CN-5 or an equivalent open comparability level should be given for the comparison between controlled and natural panels.

CH11-N47

A CN-0 or CN-1 panel difference cannot be presented as a causal personalisation effect.

CH11-N48

The panel status lock and the comparison decision must have an accountable human or institution owner.

72. FORMS OF FAILURE

CH11-F01 — CONSIDERING A NEW CONVERSATION AS A CLEAN ACCOUNT

Persistent memory and special instructions are ignored.

CH11-F02 — BLIND TRUST IN TEMPORARY SESSION TAG

It is assumed that the product label disables all personalisation until verified.

CH11-F03 — CONSIDERING UNKNOWN MEMORY AS CLOSED

The clean score is artificially increased.

CH11-F04 — IGNORING SPECIAL INSTRUCTION

Response format or recommendation guidance is presented as a product default.

CH11-F05 — CONSIDERING THE CORPORATE WORK AREA AS A GENERAL PRODUCT

Private information sources and administrator settings remain invisible.

CH11-F06 — MIXING PLANS ON A CLEAN PANEL

Free and paid users are presented as a single standard state.

CH11-F07 — CONSIDERING CLOSING TOOLS AS CLEANING

The product's natural information mode is artificially altered.

CH11-F08 — NOT SAVING TOOL STATUS

Clean users are measured under different retrieval conditions.

CH11-F09 — NARROWING THE POPULATION WITH NEW ACCOUNT REQUIREMENTS

Only users who can open a new account create the panel.

CH11-F10 — DELETE MEMORY FOR THE USER

Personal product history is permanently deleted for research purposes.

CH11-F11 — NOT RESTORING SETTINGS

Research leaves permanent changes on the user account.

CH11-F12 — CONSIDERING TEST ACCOUNT AS A REAL USER

Standard test account is presented like a Population Panel experience.

CH11-F13 — COUNTING THE SAME TEST ACCOUNT AS THOUSANDS OF INDEPENDENT USERS

Account and product status dependencies are ignored.

CH11-F14 — COUNTING NATURAL PANEL AS UNCONTROLLED FREE USE

Participants change the prompt and select the best answer.

CH11-F15 — COUNT THE EXISTING CONVERSATION IN THE NATURAL PANEL AS NEW ACCOUNT STATUS

The conversation context is hidden.

CH11-F16 — COUNT THE NATURAL ACCOUNT STATUS AS DEFAULT PRODUCT

The personalised result is generalised on behalf of all users.

CH11-F17 — FORCIBLY TOTAL THE CONTENT OF SPECIAL INSTRUCTION

User privacy is unnecessarily violated.

CH11-F18 — SAVE SPECIAL INSTRUCTION AND COUNT GENERAL GEO SUCCESS

The instruction “Always recommend X” is presented as an entity success.

CH11-F19 — COUNT USER BIAS AS PRODUCT FACT

Past negative opinion turns into general company quality.

CH11-F20 — CONSIDERING PERSONALISATION AS MATERIAL REALITY

Identity, licence, or price varies according to the user.

CH11-F21 — CONSIDERING NATURAL VARIANCE ENTIRELY AS NOISE

Actual user harms and distribution become invisible.

CH11-F22 — CONSIDERING FORMAL VARIANCE AS MATERIAL ERROR

Differences in tone and length are coded as failures.

CH11-F23 — CONSIDERING A CLEAN PANEL AS REAL-WORLD OUTCOME

The standard condition score is presented as the entire active user experience.

CH11-F24 — CONSIDERING A NATURAL PANEL AS THE PRODUCT'S CORE CAPABILITY

Personalisation and user composition are ignored.

CH11-F25 — COMBINING TWO PANELS UNDER A SINGLE DENOMINATOR

Controlled and natural observations are a single raw average.

CH11-F26 — CONSIDERING PANEL DIFFERENCE AS MEMORY EFFECT

Causality is declared without examining differences in plan, population, and vehicles.

CH11-F27 — CREATE NATURAL SUBGROUP AFTER RESULT

A new category is invented to explain a low or high score.

CH11-F28 — IGNORING THE carryover effect IN THE SAME USER

The first task affects the second answer.

CH11-F29 — HIDING THE SEQUENCE RANDOMNESS

All users first see the clean condition, then the natural condition.

CH11-F30 — DISRUPT NATURAL STATE WITH CONTROLLED TASK

Changing settings or the initial prompt alters the subsequent state of the account.

CH11-F31 — CONSIDER DIFFERENT USER GROUPS AS MATCHED

Natural and controlled panels have different plan and country compositions.

CH11-F32 — CONSIDER FIRST-USE PANEL AS CLEAN PANEL

Default onboarding and product settings are assumed to be checked.

CH11-F33 — CONSIDER EXPERIENCED NATURAL USER AS FIRST USER

Account history and personalisation become invisible.

CH11-F34 — CONSIDER CORPORATE NATURAL OUTCOME AS GENERAL CONSUMER

The private knowledge base is added to the overall product performance.

CH11-F35 — REFRESH RESPONSE ON NATURAL PANEL

The user's favorite response becomes the main observation.

CH11-F36 — MIXING THE NATURAL USER JOURNEY INTO THE FIRST RESPONSE SCORE

The response formed after follow-up questions and opening resources is presented as if it is the first exposure.

CH11-F37 — CHANGING PANEL STATUS ACCORDING TO RESPONSE

Positive observations are declared clean, negative observations are declared natural.

CH11-F38 — SILENTLY TRANSFERRING CLEANING ERRORS TO NATURAL

The observation is reclassified without a necessary natural status record.

CH11-F39 — CONSIDERING THE UNKNOWN AS PRIVATE FOR PRIVACY

Unmeasured personalisation is assumed positive.

CH11-F40 — MAKING A SMALL SUBGROUP IDENTIFIABLE IN PUBLIC

Rare special instructions or a small country user is revealed.

CH11-F41 — HIDING COVERAGE BIAS IN THE NATURAL PANEL

Users willing to share their real account are counted as the entire active population.

CH11-F42 — HIDING CLEAN PANEL EXCLUSION

Users who cannot change their settings become invisible.

CH11-F43 — ACTIVE POPULATION SCORE WITHOUT ACTIVE USER DATA

The research panel is presented as if it were the actual active user distribution.

CH11-F44 — CN-1 CAUSALITY FROM DIFFERENCE

Two descriptive scores are published as personalisation effect.

CH11-F45 — PRESERVING THE CRITICAL TAIL OF PANEL DIFFERENCE

Only the average difference is shown.

CH11-F46 — ERASING THE UNKNOWN MASS OF THE NATURAL PANEL

Users whose setting or personalisation status is unknown are considered problem-free.

CH11-F47 — PRESERVING FAVOURABLE NATURAL BIAS

No correction is made when the user instruction institution shows it as large.

CH11-F48 — GENERALISING NATURAL NEGATIVE BIAS

The past opinion of a single user is represented as AI representation of the entire population.

73. AUDIT PROCEDURE

Step 1 — Separate the Research Questions of the Two Panels

Write which quantity the controlled and natural panel will predict.

Step 2 — Define the Target Populations

Separate the controlled product-eligible population from the natural active or identified user population.

Step 3 — Define the Controlled Clean Condition

The session, memory, special instructions, tools, plan, surface, locale conditions are versioned.

Step 4 — Determine the Required CSC Level

Write the minimum clean condition confidence that will be accepted for the main analysis.

Step 5 — Determine the Natural Panel Trace

It is determined which of the roles Natural Account-State, Natural Conversation-State, First-Use Managed will be used.

Step 6 — Determine the Required NSF Level

It is written by which evidence the natural state will be accepted.

Step 7 — Set Up the Panel Assignment Design

Independent group Random assignment Matched user Rotating panel is selected.

Step 8 — Write the Setting Change and Restore Procedure

A safe process is prepared for controlled panel participants.

Step 9 — Lock the Privacy Boundary

Which natural user situations:

• only status,
• limited metadata,
• open content

it is determined that it will be saved as.

Step 10 — Lock Prompt and Initial Output Rule

The same prompt and initial appropriate output protocol are applied in both panels.

Step 11 — Save Tool and Web Mode

Controlled and natural panel differences are made visible.

Step 12 — Verify Panel Status Before Field

Is it suitable for the participant panel role?

Step 13 — Save Observation-Level Panel Vector

Memory, instruction, plan, surface, context, and usage status are stored.

Step 14 — Blind Review Protocol Validity

The panel status decision is made without seeing whether the response is correct or incorrect.

Step 15 — Manage Misclassifications with Predefined Rule

Transition from clean to natural or mixed-state is done only if the record is sufficient.

Step 16 — Generate Panel Weights

Separate weights are calculated according to the different target populations of the two panels.

Step 17 — Calculate Main Scores

Controlled representation Natural representation Critical error

UNKNOWN

Rejection and inconclusiveness are generated separately.

Step 18 — Calculate the Clean–Natural Difference

Signed and absolute difference is taken.

Step 19 — Examine Subgroups

Predefined groups such as Memory, Instruction, Plan, Account age, Usage frequency, First-use are evaluated.

Step 20 — Set the Causality Limit

The CN level is assigned. It is written which claims can be established.

Step 21 — Check Privacy and Small Cell Control

Results with a risk of re-identification are combined or restricted.

Step 22 — Create the Public Panel Scorecard

The two panels are published along with their scope and limitations.

74. NECESSARY EVIDENCE

Controlled-panel research question; Natural-panel research question; target populations of both panels; panel-assignment method; Controlled Clean-state definition; minimum CSC level; new-chat record; memory state; custom-instruction state; prior-conversation state; enterprise-workspace state; tool and web mode; plan; user surface; language and locale; setting-change instructions; participant consent; setting-restoration record; use of a test account, if any; Natural-panel trace; minimum NSF level; account-age band; frequency of use; First-Use status; brand familiarity; prior-conversation status concerning the audited entity; Natural tool state; distinction between Natural Account-State and Natural Conversation-State; prompt ID and version; first-eligible-output validation; panel-state vector; and panel-state validity status.

Validity review Reclassification records Controlled panel exclusions Natural panel coverage gaps Panel weights Raw and effective sample Controlled score Natural score Signed panel difference Absolute panel difference Critical error difference UNKNOWN and inconclusive difference Predefined subgroup results Matched design records Order and carryover effect CN comparability level Privacy and redaction record Small cell protection Public panel result card Panel version and change record Responsible person or institution

75. AUDIT CONTROL LIST

Are the research questions for the controlled and natural panel separate? Are their target populations the same or different? Was the controlled clean condition clearly defined? Apart from the new chat, which clean conditions were confirmed? Is memory really off, or is it unknown? Was the special instruction status recorded? Is there an effect of the corporate workspace? Are the web and tool modes standard? Is the plan and user surface fixed or layered? Was a confidence level given for the clean condition? Were CSC-1 observations used as fully clean? Was the participant asked to erase their memory or history? Were settings restored at the end of the task? Was a new account or test account used? Was the test account presented like a real user panel?

Which trace does the natural panel use? Has a new chat been opened in Natural Account-State? Did the current conversation context interfere with the main natural score? Was a natural state confidence level given? Were memory and custom instruction contents collected unnecessarily? Was the user's actual plan and surface recorded? Are First-Use users separate? Are enterprise users separate? Was the natural tool state recorded? Did the two panels use the same prompt version? Was the first eligible output preserved in both panels? Did the natural user response refresh? Was the panel status changed according to the AI response content? How were records with detected natural states managed in the clean panel? Was the same user used in two panels? Was the order randomised?

Was the effect of transfer and memory evaluated? In the independent group design, does the user composition match? Are controlled and natural weights separate? Can the natural score really be generalised to the active user population? Was the clean–natural difference overinterpreted as a personalisation effect? Is a critical error difference visible? Is there a visible difference for UNKNOWN and null results? Were subgroups defined before the results? Can small subgroups be redefined? Were user-led instructions for or against converted to general GEO? Did personalisation alter material reality? Was a CN comparability level provided? Is the accountable owner of the panel results card clear?

76. OBJECTIONS AND RESPONSES

Objection 1 — "If real users use memory accounts, why is a clean panel necessary?"

Because the source of the difference cannot be understood without a common starting condition between products, countries, and languages. A clean panel does not replace a real user panel. It provides a reference for comparison.

Objection 2 — "Isn't a controlled panel artificial if there is a natural panel?"

It is controlled. This is intentional. The purpose of a laboratory condition is not to be the entirety of real life, but to isolate certain variables.

Objection 3 — "If the controlled panel scores higher, can't we make that the main GEO score?"

You cannot do it alone. A high controlled score cannot erase the fact that real users see lower results. The two results should appear separately.

Objection 4 — "Is the scientific value low if the natural panel is very scattered?"

Heterogeneity does not have to be a defect of the natural panel. It is a feature of the real user experience. Heterogeneity:

• is recorded,
• is layered,
• is weighted,

is reported with uncertainty.

Objection 5 — "Doesn't opening a new chat disrupt the natural user state?"

The current conversation context is removed. The persistent account, plan, memory, and custom instruction state are preserved. This makes the natural account state comparable. The current conversation experience is also measurable.

Objection 6 — "How will we measure its effect without knowing the content of the custom instructions?"

Knowing the content is not required for the main natural score. Its presence:

• exists,
• does not exist,
• may be material

we can classify it as such. Separate and consensual work is required for causal content analysis.

Objection 7 — “If the memory is clear but nothing is remembered about the company, why a separate group?”

The effect of memory may be unknown. A status of “unrelated to the entity” can be given based on the user’s statement. This still does not have the same certainty as the closed memory condition.

Objection 8 — “Isn’t the user’s ‘always suggest X’ instruction real user experience?”

It is real experience. However, it is not the general GEO performance of X. It is a user-sourced guidance. It should be recorded separately.

Objection 9 — 'If personalisation gives a more suitable answer to the user, what's the problem?'

It does not have to be a problem. Recommendations may vary according to the user's need. Problems occur if identity, licence, price, and evidential status change arbitrarily.

Objection 10 — “Isn't using the same user in two panels the most correct method?”

It can be strong for some causal comparisons. The first task can affect the second situation. Therefore, order, carryover, and memory control are needed. Independent groups may be safer in main population panels.

Objection 11 — 'If refreshing the answer in the natural user panel is normal behaviour, why is it forbidden?'

Regeneration is prohibited when measuring exposure to the first response. Regeneration and follow-up behaviour may be measured separately under a Natural Interaction Track.

Objection 12 — “What will we do with a user who cannot turn off memory and instructions for a clean panel?”

They cannot be taken to the controlled panel or are kept in a separate cell with a lower CSC level. They can be evaluated in the natural panel. The exclusion rate should be reported.

Objection 13 — “Aren’t test accounts more reliable?”

They may be reliable in terms of control. They are limited in terms of the actual user population. Controlled test accounts and real user panels are separate results.

Objection 14 — “Isn’t saving user settings on the natural panel a privacy violation?”

It may happen if unnecessary content is collected. Privacy can be preserved with high-level open/closed statuses, anonymous registration, and redaction.

Appeal 15 — “Aren't we going to combine the scores of the two panels into a single NOMOS score?”

The roles of the two panels can be defined in the final score architecture. However, raw and sub-results should always remain visible separately. They cannot replace the combined score distribution.

Appeal 16 — “What fault does the company have if the natural panel scores low?”

It is not always the company's fault. Source:

• user instruction,
• AI personalisation,
• plan difference,
• official source defect,
• product behaviour

it is possible. Audit first shows the result, then separates the path of responsibility.

COMMON RULE OF CHAPTER 81

An AI product does not have a single answer. Even the same prompt can have different answers for:

• clean account,
• natural account,
• user with memory,
• user with special instructions,
• first-time user,
• corporate workspace,
• user with previous conversation

You cannot explain all these differences by saying: “the model is behaving randomly.” You also cannot explain all of them by saying: “personalisation error.” First, you need to measure two separate realities. Controlled Clean Panel:

Shows the representation in the system's standardised initial condition.

Natural User Panel:

Shows the distribution of representation encountered by real people in their own product life.

Controlled panel may be high, natural panel may be low. In this case, the product's laboratory behaviour is strong; the real user distribution may be weak. Natural panel may be high, controlled panel may be low. Some users' planning, tools, or personalisation may be improving representation. Both panels may be high. The product may be robust in different user conditions. Both may be low. The fundamental representation problem may be broader. However, placing the two scores side by side does not prove the reason. Causality:

• same population,
• same product,
• same prompt,
• same time,
• controlled assignment,
• sequence and transport analysis

is required. If a user has instructed: “Always suggest X.”, it is natural experience for X to appear. X is not a global GEO success. If a user says: “Do not trust X.”, a negative response is natural experience. X is not a general representation in the world population. Personalisation should understand the user. It should not reinvent the truth according to the user. Therefore, NOMOS's eleventh measurement law is:

Clean behaviour and natural experience are separate realities.

The twelfth law is as follows:

A new chat is not a clean account.

The thirteenth law is as follows:

Natural personalisation is not a global GEO success.

The fourteenth law is as follows:

The appropriate response for the user may vary; the reality of the verified entity cannot change.

The fifteenth law is as follows:

The difference between controlled and natural score is the finding that needs to be explained; it is not a reason by itself.

Its sixteenth law is as follows:

Measuring the real user experience is not about removing privacy, but only proving the necessary context and preserving the rest.

Order of Section 11 of NOMOS

Do not just tell me that you opened a new chat. / Show the status of memory, special instructions, plan, and tools.

Do not assume that all personalisation is gone just because you see the note 'Temporary session'.

The clean panel is the entirety of real life; do not make the natural panel uncontrolled noise.

Standardise the controlled account. / Keep the natural account as it is. / Do not put both in the same box.

Do not make the user's special instructions the world's success for the entity.

Do not make the user's negative past the view of the entire population.

Do not count personalisation as a licence to change the truth.

You can give a user suitable advice. / But do not rewrite the company's identity, licence, price, and proof from user to user.

Do not collect private life in the natural panel. / Is memory on, instructions present, context available — only record what is necessary.

Do not make the participant erase their memory. / Restore the settings if you temporarily changed them.

If you tested the same user on two panels, remember that the first answer may affect the second one.

Show the difference between the clean score and the natural score. / Show the critical difference. / Show the unknown. / But do not name the reason without proof.

First, decide which user reality you want to measure. / Then lock the clean and natural conditions separately. / Then keep the prompt the same. / Then capture the first answer without changing it. / Then calculate the two distributions separately. / And only after this, explain the distance between what the product does under standard conditions and what real people experience.

The Chapter's Closing Sentence

The true representation of an AI product in GEO-1000 is not the answer it can provide in a clean session; it is the clear and proven relationship between its behaviour under standard conditions and the answers it actually delivers to people across different user histories.

Normative Core

Controlled Clean Panel and Natural User Panel observations MUST remain separate measurement populations, panel states, denominators, and public results. A Controlled Clean Panel MUST define and verify, as applicable: - a new and separate conversation, - absence of prior conversation context, - persistent-memory state, - custom-instruction state, - enterprise or managed-workspace state, - web, retrieval, and tool configuration, - plan, - user surface, - prompt version, - and clean-state confidence. A new conversation or temporary-session label MUST NOT, by itself, be treated as proof that all persistent personalisation has been disabled. A Natural User Panel MUST preserve the participant's authentic account, plan, interface, memory, custom-instruction, and usage state while retaining the locked prompt, first-eligible-output rule, evidence requirements, and non-regeneration rule. Natural Account-State, Natural Conversation-State, First-Use, and Managed Natural observations MUST remain separately identified. Natural-state documentation MUST minimise private-data collection. Private instruction text, conversation history, identity, credentials, and sensitive personal content MUST NOT be collected unless separately necessary, consented, minimised, and access-controlled. User-generated entity-favourable or entity-adverse instructions MAY be measured as natural personalisation effects, but MUST NOT be represented as general entity GEO success, failure, or default product behaviour. Personalisation MAY change relevance, presentation, and recommendation where user context justifies it. It MUST NOT arbitrarily change verified entity identity, licence, legal status, price, scope, evidence, or time. Differences between Controlled and Natural panels MUST NOT be represented as causal personalisation effects unless population, product, prompt, time, interface, plan, tools, assignment, order, and carryover conditions support that inference. Controlled and Natural scores, critical-error rates, unknown rates, coverage, effective sample sizes, and signed and absolute panel gaps MUST remain separately visible. Panel-state eligibility, reclassification, exclusion, and weighting MUST remain independent of whether the AI output is positive, negative, correct, incorrect, refused, critical, or commercially favourable. Every panel-state definition, setting-change process, privacy decision, comparison level, and result MUST be versioned and attributable to an accountable human or organisation.