The Population-Proportional GEO-1000 Sample

Chapter Boundary

Section 5 established the following provision:

The target user population of each AI product must be defined separately. The total population of a country is not the product-eligible user population in that country.

Now we come to the next question:

From the defined eligible user population for GEO-1000, how many people will be selected, from which country, which language, and by which method?

The eligible user population for an AI product is:

• 40% in country A,
• 25% in country B,
• 15% in country C,
• 0.04% in a small country

If available, how should 1,000 valid observations be distributed? How will fractional quotas be converted into whole numbers? Will countries with a population share of less than one person be completely excluded? Will population weighting be distorted when at least one person is given to each country? Can small countries be grouped into a separate pool? If some languages are oversampled for language fairness, how will the global score be calculated? If the target is 100 valid observations in a country, how many people should be invited? Who will replace the participant if they do not complete the task? Can a user giving wrong answers be replaced? If the same person tests more than one AI product, will the sample still be independent?

Even with a raw sample of 1,000 people, can highly unequal weights reduce the effective statistical information below that of 1,000 independent, equally weighted observations? These are not merely operational details. The sample design determines how much influence each person has on the GEO score. A defective sample:

• can misweight the correct answers,
• can symbolically overrepresent small countries,
• can make large populations appear smaller than they are,
• can make low-resource languages invisible,
• can present voluntary technology users as if they were the world population.

It can be adjusted to achieve the desired score after reviewing the results. The previously established audit architecture requires that each audit be measurable and that the model/product carry the date, country, language, query set, repetition count, and measurement record. This section applies the same discipline to sample design. Because genuine evidence requires requesting limits, context, and time; it also necessitates making how the sample was created visible. This section:

• how the 1,000 valid observations of the main population panel will be distributed,
• how strata and cells will be established,
• how small countries will be managed,
• how fractions will be converted to whole numbers,
• how oversampling will be weighted,
• how to separate invitation, backup, and valid observation targets,
• how to establish design and analysis weights,
• why the effective sample size may differ from the raw number

defines. This chapter does not yet:

• from which panel provider the participants will be selected,
• the full details of field implementation in voluntary or probability sampling,
• reviewing and scoring responses,
• the final confidence interval method,
• all NOMOS scoring formulas

are not fully finalised. The task of Section 6 is:

Transform the defined target population into a frozen, weightable, and reproducible GEO-1000 sample before seeing the outcome.

NOMOS Challenge

You are setting up a global panel of 1,000 people for an AI product. The product-eligible populations of ten countries are as follows:

Raw quotas according to pure population proportion:

Half of a person cannot be selected. 0.4 people cannot be sent to a country. If you round the raw quotas to the nearest whole number, the total may exceed or fall short of 1,000. If you give at least one person to each country, you over-represent countries I and J compared to their population shares.

Giving one person to country I increases its theoretical share 2.5 times. Giving one person to country J increases its theoretical share 10 times. This extreme representation can largely be corrected. But you cannot produce a country score with a single person. What happens if you give no one to I and J?

If you define these countries as separate and mandatory strata, the probability of some people in the target population being selected becomes zero. In this case, the claim of a “probabilistic sample representing the entire appropriate global population” weakens. Now consider a more dangerous decision. First, you collect the answers.

The only user in country J gets an incorrect answer. Then you say: “The sample size of this country is very small, let's exclude it from the scope.” In country A, there are many positive answers. You retain it with full weight. This is no longer rounding. It is a result-oriented sample change.

In another case, you increase the invalid record rate in a language group with low results, using backup participants only until a positive response is obtained. This is also not a sample. It is a response selection mechanism. The first ruling of this section is:

The sample must be established before seeing AI responses.

Its second provision states:

The probability of a participant being selected cannot depend on the correctness of their answer.

Its third provision states:

The visibility of the small country and the population weight in the global score must be managed separately.

Its fourth provision states:

Oversampling is possible; over-weighting is not mandatory.

Its fifth provision states:

1,000 raw records do not always carry statistical information equivalent to 1,000 observations.

1. PURPOSE OF THE CHAPTER

The purpose of this section is to transform the targeted 1,000 valid observations for each AI product and measurement wave into a verifiable sample design based on the product-eligible user population. The section standardises the following distinctions:

• Target population versus selected sample
• Raw quota versus integer allocation
• Main population panel versus complementary panels
• Country visibility versus country weight
• Strata versus experimental cell
• Large country strata versus small country pool
• Proportional allocation vs. disproportionate oversampling
• Sample share vs. population share
• Number of invitations vs. valid observation target
• Backup participant vs. post-result replacement
• Data collection invalidity vs. adverse AI output
• Design weight vs. outcome weight
• Raw sample size vs. effective sample size
• Country sample vs. language sample
• Multilingual user vs. more than one person
• Single-stage selection vs. multi-stage selection
• Native Reach allocation vs. Common Support allocation
• Instant wave allocation and time series allocation
• Population estimation and country or language recognition analysis
• Main 1,000 observations with auxiliary observers and fairness panels

At the end of this section, each audit should be able to answer these questions:

Why were 1,000 observations distributed in these numbers across this country, language, and user groups?

How was the weight of each observation in the global result determined?

Who had a zero probability of being selected and why?

How many people were invited, how many completed, and how many observations remained valid?

Were the sample results frozen before being viewed?

2. CENTRAL NORMATIVE PROVISION

The GEO-1000 main population sample should be based on a stratified sampling design that assigns known or predictable probabilities of selection to people in the pre-frozen product-eligible population frame for each AI product and measurement wave. The default target of the main population panel is:

At least 1,000 valid observations per AI product and main measurement wave

is maintained. This number:

• is not the number of people invited,
• is not the number of accounts,
• is not the number of raw screenshots,

It is not the sum of all supplementary panels. 1,000 is the target for valid product-observations meeting the protocol in the main population panel. Country Observer, Language Fairness, and Accessibility panels can create additional samples. These additional samples:

• can increase the total field volume above 1,000,
• can be used with appropriate weights in the global population outcome,

but cannot be used to covertly reduce the 1,000 valid observation target of the main panel.

3. BASIC REPRESENTATION OF THE SAMPLE

For a specific AI product a, measurement wave w, and target population frame, the strata:

h ∈ H_{a,w}

are denoted. For each stratum:

• Nh: estimated product-appropriate population in the stratum
• N: total target population
• Wh: population weight of the stratum
• n: target valid main panel observation
• qh: crude proportional quota
• nh: allocated integer valid observation target

is. Total target population:

N = Σ_{h=1}^H N_h

Population weight of the stratum:

W_h = N_h / N

Crude proportional quota:

q_h = nW_h

is. Main proportional allocation, under ideal conditions:

n_h ≈ nW_h

should be. However:

• qh may be fractional,
• some cells may fall below one person,
• a minimum diagnostic sample may be required for language and country subgroups,
• high-risk small groups may also be examined,

field completion rates may vary. Therefore, the raw quota is not directly the final allocation.

4. THREE DISTINCT PURPOSES OF SAMPLE DESIGN

The same sample may not fulfil all purposes with the same strength. Three purposes should be separated.

4.1. Global Population Estimate

Answers the question:

What representative outcome would the average person in the global user population suitable for the product see?

Population weight is the primary priority.

4.2. Subgroup Diagnosis

Answers the question:

What representation distortions occur in specific countries, languages, plans, or user groups?

It may be necessary to oversample small groups.

4.3. Geographic and Linguistic Coverage

Answers the question:

In which countries and languages is there at least real user observation?

Country Observer and Language Fairness panels become important. These three goals cannot be inexplicably combined in a single sample.

5. MAIN SAMPLE ARCHITECTURE

The candidate main sample architecture for GEO-1000 consists of four layers.

5.1. Population Panel

Every AI product and wave has a minimum of 1,000 valid observations as the main panel. It produces a global population estimate.

5.2. Country Observer Overlay

Provides additional geographic observations for small countries that cannot be directly represented in the main population panel. These observations can indicate:

• single event,
• access status,
• future research needs

They do not generate a country score on their own.

5.3. Language Fairness Oversample

Generates additional observations for low-resource languages or languages underrepresented in the main panel. They are reweighted to actual population weights in the global outcome.

5.4. Accessibility Oversample

It also measures users of assistive technology and accessibility, who are not adequately represented by the standard panel. The target universe and weight of these observations are recorded separately.

6. WHAT IS A STRATUM?

A stratum is a segment of the population that is managed separately in terms of sample allocation and selection probability. The stratum can be a combination of one or more of the following areas:

• Country
• Region
• Language
• Locale
• AI plan
• Web or mobile
• Controlled or natural user panel
• Accessibility status
• User type
• Jurisdiction

If each small combination is made a separate stratum, the sample is fragmented. For example: Country × language × plan × mobile/web × age group × user type combination can produce hundreds or thousands of cells. 1,000 observations do not provide reliable estimates for most of these cells. Therefore, stratification should be:

• substantial,
• measurable,
• appropriate for sample size

must be.

7. A STRATUM IS NOT THE SAME AS A REPORTING CELL

It can be shown separately on a group result card. However, it does not have to be a separate stratum in sample design. For example:

• a small language,
• a specific accessibility group,
• a certain age band

It can be selected within a broader stratum in the main sample. Later, it can be reported separately in the analysis. The reverse is also possible: Two regions may be separately stratified for sampling efficiency. However, in the public report, they can be combined under a single regional result. The stratification design and the reporting structure should carry separate records.

8. SEQUENCE OF STRATUM ESTABLISHMENT

The candidate sequence for the Main Population Panel is as follows:

• AI product and user surface
• Native Reach or Common Support population frame
• Country or small country pool
• Main audit language or language group
• Material plan/interface separation, if necessary
• Participant selection frame

The plan and interface do not have to be made into separate layers for each country. These can be managed through:

• sub-layer,
• quota,
• random assignment

methods.

9. LARGE AND SMALL COUNTRY LAYERS

A country's raw quota:

q_c = nN_c / N

is in this form. If the raw quota is large enough, the country can be kept directly as a separate layer. This can be called:

self-representing country layer

Making countries with very small raw quotas individually into separate mandatory layers may reduce sampling efficiency. These countries can be appropriately:

• regional,
• language family,
• product access regime,
• geographical proximity

can be included in small country pools.

10. SMALL COUNTRY POOL

A small country pool:

Let it be represented as B_r = {c_1, c_2, …, c_k}

The total eligible population of the pool:

N_{B_r} = Σ_{c∈B_r} N_c

Main sample quota:

q_{B_r} = nN_{B_r} / N

is calculated as. Within the pool, country selection can be done with probability proportional to the eligible population size. The probability of country c being selected within the pool:

Pr(c | B_r) = N_c / N_{B_r}

It may be. Then a participant is selected within the chosen country. This approach:

• can prevent small countries from having a zero chance of being selected in the target population,
• does not guarantee that each country will be selected in a given wave,

preserves the global population estimate. Not selecting a country in that wave:

NOT SELECTED IN POPULATION SAMPLE

does not mean: "The country is not part of the target population."

11. LIMITATIONS OF POOLING SMALL COUNTRIES

Small countries cannot be randomly placed in the same pool just because their populations are small. Countries within the pool are similar to the extent that they are relevant:

• product access,
• user surface,
• language or locale structure,
• legal access,
• geographical region,
• data collection method

must be carried. A country without product access cannot be placed in the same main pool as a country with full access. Different usage conditions cannot be made invisible.

12. SMALL COUNTRY POOL DOES NOT PRODUCE COUNTRY SCORE

One or two observations selected from the pool: can be used in the global population contribution of the pool, may indicate a single country event. However, it does not produce a separate statistical score for a small country. If a country-based score is wanted, additionally:

• Country Observer,
• country-specific sample,
• multi-wave cumulative sample

may be required.

13. ALLOCATION PROPORTIONAL TO PURE POPULATION

Pure proportional allocation: is based on the formula:

q_h = nW_h

Advantages:

• it is understandable for the main population estimate,
• it naturally gives weight to large populations,
• it can reduce weight differences,

it can approach a self-weighting design. Limitations:

• it can sample very few small countries and languages,
• it may be insufficient to produce subgroup scores,
• it may leave high-risk small groups invisible,

it can create fractional and near-zero quotas. Therefore, pure proportional allocation can be a strong default for the main Population Panel. It does not replace complementary panels.

14. CONVERTING FRACTIONS TO WHOLE NUMBERS

Raw quotas are often fractional. The total allocation should be exactly n. The Largest Remainder Method can be used as the default method.

14.1. First Step

Each raw quota is rounded down:

b_h = ⌊q_h⌋

14.2. Second Step

The number of unallocated observations:

R = n − Σ_h b_h

is calculated.

14.3. Third Step

The fractional part of each stratum:

f_h = q_h − b_h

where f_h is the fractional part of q_h.

14.4. Fourth Step

The remaining R observations are added one by one to the layers with the highest fh values.

14.5. Equality Situation

If the fractional parts of the two layers are equal:

• population size,
• predetermined fixed order,
• registered random seed

An independent tie-breaking rule should be used, regardless of the outcome. A country cannot be chosen based on the results.

15. SYNTHETIC INTEGER ALLOCATION

SYNTHETIC DISPLAY

Appropriate populations at the beginning of the section:

If countries are protected as separate layers, round down:

One observation remains. It is added to H because the highest fraction is 0.5 in country H. Final allocation:

Even though this design defines I and J as separate deterministic strata, if it does not give a chance to be selected, it may pose a problem in terms of full target universe inference. A more accurate solution is a small country pool.

16. SYNTHETIC ALLOCATION WITH SMALL COUNTRY POOL

Countries H, I, and J:

B_small = {H, I, J}

should be combined in the pool. Pool population:

1,500,000+400,000+100,000=2,000,000

Pool quota:

1,000 × 2,000,000 / 1,000,000,000 = 2

becomes. Main allocation:

Selection weights of countries within the pool:

Two main population observations are selected according to these probabilities. In the same wave:

• both observations can come from H,
• one from H and one from I,
• rarely from J

This can occur. The design does not make J visible in every wave, but it does not reduce J's probability of selection to zero. If J must be observed, it is added separately to the Country Observer Panel. A Country Observer observation does not give J a weight of one in 1,000 in the main global allocation; J retains its true population weight in the main panel.

17. MINIMUM CELL ASSIGNMENT

Purely proportional quota may not be sufficient to perform diagnostic analysis for some subgroups. For example, the raw quota for a language cell might be 8. A reliable language score cannot be produced with eight observations. In this case, there are three options:

• Not to publish the language score
• Combine the cell with similar groups
• Oversampling the cell

Minimum cell allocation must be defined before the results.

18. CANDIDATE REPORTING BANDS

CANDIDATE REPORTING BANDS — CANDIDATE REPORTING BANDS

The thresholds below are not final scientific decisions. They should be calibrated through pilot and external review.

1–29 Valid Observation

Single event and discovery Country or language rate is not published

OBSERVATIONAL ONLY

30–99 Valid Observations

Exploratory level estimate Broad uncertainty No definitive conformity judgement is given

EXPLORATORY

100–199 Valid Observations

First subgroup estimate Uncertainty should be visible

PROVISIONAL

200 and Above Valid Observations

Stronger subgroup estimate candidate Design effect and weights are again evaluated

STANDARD ESTIMATION CANDIDATE

These bands:

• error rate,
• expected precision,
• weight variability,
• design effect

cannot be automatically considered sufficient without being taken into account.

19. OVERSAMPLING

A country, language, or user group may have more observations than its population share. This is called oversampling. Its purposes:

• to increase subgroup sensitivity,
• to make a low-resource language visible,
• to examine a high-risk small group,
• to expand Country Observer coverage,
• to adequately represent accessibility users

are possible. Oversampling is not wrong. What is wrong:

It is to use the sample proportion of the over-sampled group as if it were the true population proportion.

20. POPULATION SHARE AND SAMPLE SHARE

Population share for layer h:

W_h = N_h / N

Sample share:

S_h = n_h / n

is in the form. In pure proportional allocation:

S_h ≈ W_h

okay. In excessive sampling:

S_h > W_h

It is possible. In a small sample:

S_h < W_h

It is possible. The global population estimate should retain Wh values instead of Sh.

21. BASIC DESIGN LOAD

In simple stratified sampling, the basic weight of each observation within stratum h can be considered as:

d_i = N_h / n_h

The normalised stratum weight can be written as:

a_h = W_h / S_h

The normalised observation-level weight can be:

w_i = W_h / n_h

It is ensured that the sum equals one:

Σ_i w_i = 1

The weighted binary transition estimate can be calculated as:

θ̂ = Σ_i w_iY_i

Here:

Yi=1: response meeting the transition criterion

Yi=0: response not meeting it

It is possible. The final NOMOS system does not have to use only binary outcomes. The same weighting logic can also be applied to multidimensional response scores.

22. EXCESS SAMPLING EXAMPLE

SYNTHETIC EXAMPLE

In the suitable population of country B:

• Language X: 80 per cent
• Language Y: 20 per cent

Let it be. In the Population Panel, there are 250 observations for country B. Pure allocation:

• Language X: 200
• Language Y: 50

Let 50 additional Language Fairness observations be taken for Language Y. Collected observations:

• Language X: 200
• Language Y: 100

The share of Language Y in the raw sample:

100/300 = 33.3%

It has been. The actual population share is 20%. In the global population account, Language Y cannot take a 33.3% weight. Within-stratum normalised weights:

• Total weight for Language X: 80%
• Total weight for Language Y: 20%

is preserved. Each of the 100 observations in Language Y receives a lower individual weight than a single observation in Language X. In contrast, all 100 observations are used in Language Y’s separate diagnostic report. This approach: increases subgroup visibility, does not distort the global population estimate.

23. ALLOCATION OF MULTILINGUAL USERS

A person may be suitable for more than one language cell. One of the preferred approaches for the main Population Panel is to assign a predefined main audit language to the user. The main audit language can be determined according to these fields:

• Daily AI usage language
• Mother or home language
• Market and locale
• Competence in naturally using the prompt
• Panel balance

If the user will also be tested in other languages, these observations:

• paired multilingual panel,
• Language Fairness Panel

are also labelled accordingly. It is prohibited for the same person to receive full population weighting in all languages.

24. LANGUAGE ALLOCATION WITHIN THE COUNTRY

Let the suitable population for language groups l in country c be: N_{c,l}. If the country quota is: n_c, then the raw language quota:

q_{c,l} = n_c × N_{c,l} / Σ_l N_{c,l}

can be calculated as such. However, if language groups overlap:

Σ_l N_{c,l} > N_c

may be possible. In this case, using a direct share creates double counting. Solution options:

• main audit language classification,
• split human weight,
• multilingual common model,
• separate matched language panel

may be used. The method used should be recorded.

25. ALLOCATION OF PLAN AND INTERFACE WITHIN THE POPULATION

Free and paid users or web and mobile surfaces may behave differently. However, dividing each country–language cell separately into plan and interface layers can overly fragment the sample. Three approaches can be used.

25.1. Suballocation Proportional to Population

If the actual user distribution is known, allocation is made according to plan and interface shares.

25.2. Balanced Experimental Allocation

Equal or greater sampling is carried out to compare the difference between plan or interface. In global estimation, it is re-weighted to the actual usage shares.

25.3. Separate Product Surface Scores

Free web, paid web, and mobile products are reported as separate audit objects. A combined result can also be produced. The approach to be used should be defined before measurement.

26. INVITATION NUMBER AND VALID OBSERVATION TARGET

If the valid observation target for a stratum is nh, inviting exactly nh people is usually not sufficient. Because:

• some people do not participate,
• some do not complete the task,
• some records remain out of protocol,

some evidence packages become invalid. Let the expected completion rate for the stratum be rh and the expected validity rate be vh. The invitation target is approximately:

m_h = ⌈n_h / (r_h v_h)⌉

can be calculated as follows. Example:

• Valid observation target: 100
• Expected completion rate: 80 per cent
• Expected validity rate: 90 per cent

m_h = ⌈100 / (0.80 × 0.90)⌉ = 139

people can be invited. This is not a guarantee. As the field progresses, the predefined backup system can be used.

27. LIMIT OF INVITATION INFLATION

Inviting more people alone does not solve sample bias. Early responders may be systematically different from late responders. Only taking the first 1,000 completers:

• time zone,
• technology interest,
• connection quality,
• work schedule

may produce bias in terms of. Invitation and selection order:

• should be randomised,
• must be recorded within the stratum,

should be managed throughout the field window.

28. ALTERNATE PARTICIPANT SYSTEM

An ordered backup list should be created for each stratum before the results are seen. A backup participant can be activated only for the following reasons:

• No response to the invitation
• Failure to complete the task
• Technical or ethical suitability issue
• Violation of the demand protocol
• Invalidity of the evidence package
• Re-registration of the same person

The backup user cannot intervene for the following reason:

The previous user's AI response gave an incorrect, negative, or unwanted output.

An incorrect AI response is the valid system result. It cannot be a reason for participant change.

29. RESULT BLINDNESS IN VALIDITY DECISION

The person deciding whether the observation complies with the protocol as much as possible:

• must evaluate the response without knowing whether it is correct or incorrect,
• or whether it is positive or negative for the brand.

First:

• Capture and protocol validity
• Then the substantive response evaluation

must be done. Otherwise, negative responses can more easily be declared: "invalid."

30. RESERVE ORDER WITHIN LAYER

The reserve participant should be from the same as much as possible:

• country,
• language,
• product surface,
• plan,
• panel condition

layer. If no reserve remains in the same layer, predefined escalation is applied:

• Additional random selection within the same layer
• Predefined upper layer pool
• Leaving allocation incomplete and scope warning
• New analysis version

After the result is seen, bringing a user from another country and closing the gap cannot be done silently.

31. WHAT HAPPENS IF A LAYER IS TOO SHORT?

It may fall below a layer target. Example:

• target: 100
• valid observation: 72

Correct options:

• Extend the field window to a predefined extent
• Use the backup list
• Merge the cell with a similar and predefined upper layer
• Report the result with wide uncertainty
• Do not publish the cell result
• Recalculate the global weight using an appropriate method

Wrong option: Completely remove the low-scoring layer from the analysis.

32. POST-RESULT QUOTA CHANGE

The following changes are result-oriented sample manipulations:

• Reducing the target quota of the low-scoring country
• Increasing the quota of the high-scoring language
• Invalidating users who provide negative responses
• Redefining the small country pool based on results
• Excluding a plan with critical errors
• Adding new users until the desired average is reached

Each material quota change:

• new allocation version,
• justification,
• date,
• decision maker
• Preservation of the previous result

should be done with.

33. MULTI-STAGE SAMPLING

Multi-stage sampling can be used when direct access to all individuals is not possible. Example:

• Selection of region or country pool
• Country selection
• Selection of panel or local research partner
• User selection
• AI product or task assignment

The total probability of selection for an individual user is the product of the conditional probabilities at each stage:

π_i = π_r × π_{c|r} × π_{f|c} × π_{i|f}

Here:

• r: region or small country pool
• c: country
• f: sampling frame or panel
• i: person

Basic design weight:

d_i = 1/π_i

If selection probabilities cannot be recorded, the design cannot be presented as a fully probability-based sample. [K04; K07]

34. ALLOCATION IN VOLUNTARY AND NON-PROBABILITY PANELS

A true probability-based world panel may not always be possible. A voluntary or professional online panel can be used. In this case, country and language quotas can still be set according to the product-eligible population. However:

• the participant's actual probability of selection may be unknown,
• self-selection may occur,

weighting can only make adjustments based on observed variables. The correct status:

Non-probability user panel stratified and calibrated to the population

may be. The following statement cannot be used: “1,000 people selected completely at random from the world population.”

35. WEIGHTING ARCHITECTURE

The final observation weight may consist of multiple components:

w_i = d_i × a_i^{NR} × g_i^{CAL}

Here:

• di: base design weight
• aiNR: non-response adjustment
• giCAL: calibration coefficient

35.1. Design Weight

It is the inverse of the participant's probability of selection in the sampling design.

35.2. Nonresponse Adjustment

For selected persons with similar characteristics:

• how many responded,
• how many produced a valid observation

can be taken into account. For sample response class r:

a_r^{NR} = (Σ_{j∈S_r} d_j) / (Σ_{i∈R_r} d_i)

Here:

• Sr: selected eligible sample
• Rr: those who responded or produced a valid observation

may be. Nonresponse adjustment: does not completely resolve unobserved bias, should be explained along with its assumptions.

35.3. Calibration Weight

The sample, to known population totals:

• country,
• language,
• age,
• plan,
• user surface.

Calibration may be applied across these dimensions, but only where reliable external totals exist.

36. WEIGHT NORMALISATION

Weights:

• to the total population,
• to the total sample size,
• to individual

can be normalised. The normalisation used in the score formula should be explained. Individually normalised weights:

w̃_i = w_i / Σ_j w_j

are obtained. Weighted estimate:

θ̂ = Σ_i w̃_iY_i

is calculated as follows.

37. WEIGHT TRIMMING

Some participants may receive very high weights because they represent very small and under-sampled groups. This situation:

• can increase the estimated variance,
• the influence of a single observation,
• and score volatility

can increase. Weights can be trimmed at an upper limit. However, trimming: may create bias in population representation, may make small groups invisible again. Therefore, weight trimming should:

• be defined before the results,
• be shown together with the untrimmed results,

and carry a sensitivity analysis.

38. EFFECTIVE SAMPLE SIZE

Even though the raw number of observations is n, if the weights vary greatly, the actual amount of information may be lower. Kish effective sample size:

n_eff = (Σ_i w_i)^2 / Σ_i w_i^2 [K08]

can be shown as. With equal weights:

neff≈n

is obtained. If the weights are very unbalanced: neff<n. a GEO-1000 study:

• 1,000 valid observations,
• may carry 620 effective sample size

. To the public only: saying “1,000 people were measured.” can appear strong due to the precision. In the main report, at least:

• raw valid observation count,
• effective sample size,
• weight variability

should be determined.

39. WEIGHT DESIGN EFFECT

Approximate design effect due to weights:

DEFF_w ≈ 1 + CV(w)^2

can be shown. Here:

CV(w) = sd(w) / mean(w)

This is the coefficient of variation of the weights and only an approximate representation of their effect. The total design effect may also vary with:

• clustering,
• stratification,
• repeated measurement,
• paired users

These effects must be estimated separately.

40. CLUSTERING

The same:

• From the research panel,
• from the country,
• from the language community,
• from the device or institution,
• from the user

observations may resemble each other. In this case, 1,000 observations are not 1,000 completely independent units of information. Clustering:

• in sample design,
• in variance calculation,
• in effective sample size

should be taken into account.

41. TESTING MORE THAN ONE AI PRODUCT BY THE SAME USER

Having a user test multiple AI products allows the difference between models to be measured on the same person. Advantage: user characteristics can be controlled, paired comparison can be strengthened. Limitation:

• product observations are not independent,
• the order of tasks can affect other responses,
• a user may carry the answer seen in the first system to the next system,

users who can access all products may differ from the population. Therefore, products:

• should be assigned in random order,
• previous answer sharing should be prevented,
• the paired design should be recorded,

Analysis should protect dependency. Every AI product should again have its own target of 1,000 valid product-observations. If the same 1,000 humans test ten products:

• 1,000 unique humans,
• 10,000 product-observations

can be formed. These two numbers are not the same.

42. BALANCED MISSING BLOCK DESIGN

Not all users need to access all AI products. For example, if there are ten AI products, each participant can be assigned three products. If assignments are made in a balanced way:

• each product 1,000 observations,
• each product pair sufficient shared users,
• broader population participation

can be achieved. As a candidate for this approach:

Balanced incomplete block design

can be used. Its advantages:

• reduces participant burden,
• lowers the risk of selecting only heavy AI users,

provides matched comparison across models. The exact design will be detailed in Sections 8 and 9.

43. NATIVE REACH ALLOCATION

Each AI product receives a separate allocation according to its own Native Reach population. Therefore, the same country:

• 200 for one product,
• 0 for another product,
• 50 for a third product

can be observed. This difference is not unfairness. It is that the real access universes of the products are different. Since Native Reach scores belong to different populations, they should not be used alone for direct ranking.

44. COMMON SUPPORT ALLOCATION

A common access universe is created for comparison between products. The common population:

U_CS = ⋂_{a∈A} U_a

can be considered in this way. Each product is measured with the same Common Support population weights. This approach:

• provides fair product comparison,
• same country and language floor

ensures. However, it does not show the natural global access difference of the products. Therefore, Native Reach and Common Support results should be reported together.

45. MANAGEMENT OF THE SAMPLE ACROSS WAVES

There may be two different objectives across multiple measurement waves.

45.1. Current Cross-Section Result

In each wave, a current product-eligible population frame is used. This score answers the question: "What is happening now in the actual global access universe?"

45.2. Fixed Frame Trend Result

The population weights from the initial wave are preserved. This score answers the question: "How has representation in the same reference population changed over time?"

45.3. Separation of Two Results

When the product expands to new countries, the current score may decrease. This decrease:

• is due to the deterioration of the product,
• from the inclusion of new and difficult populations

It may be attributable. A trend with a fixed frame can show improvement at the same time. Therefore:

• Current-Frame Snapshot
• Fixed-Frame Trend

They are separate results.

46. PANEL RENEWAL AND TRANSFORMATION

Among the waves:

• the same users,
• completely new users,
• partially renewed panel

usable.

46.1. Full Section Panel

New users are selected in each wave. It can be strong for population estimation. It does not measure individual change.

46.2. Longitudinal Panel

The same users are measured again. Shows individual user and product changes. The panel may create wear and learning effects.

46.3. Rotating Panel

A portion of the participants is retained, while a portion is renewed. Provides both trend and fresh population representation. The panel structure used should be visible on the result card.

47. SAMPLE LOCK

The following records should be locked before each wave starts:

• AI product
• User surface
• Target population version
• Native Reach countries
• Common Support countries
• Strata
• Small country pools
• Raw quotas
• Integer allocations
• Language allocations
• Oversampling plan
• Invitation targets
• Reserve lists
• Selection method
• Weight formula
• Result blindness rule
• Shortfall escalation
• Analysis plan
• Decoding method
• Random seed, if used
• Responsible human approval

To this file:

NOMOS Sample Allocation Lock

can be given the name.

48. CHANGES ALLOWED AFTER SAMPLE LOCK

The following events may require a new version:

• Termination of product access in the country
• Serious field safety issue
• Material error in population data
• Panel provider unable to deliver to a specific country
• Change in research ethics requirement
• Critical error in measurement tool

Change:

• must be justified,
• dated,
• approved,
• carried out while preserving the old version.

The strength or weakness of the results is not a reason for change.

49. SAMPLE QUALITY STATUSES

A sample wave can carry one of the following statuses:

SQ-0 — NOT DESIGNED

There is no documented sample design.

SQ-1 — CONVENIENCE OBSERVATION

There is convenience or voluntary observation. Population generalisation cannot be made.

SQ-2 — QUOTA-CONTROLLED

There are country and language quotas. Selection probabilities are not fully known.

SQ-3 — WEIGHTED PANEL

There is a stratified panel and population calibration. Coverage and self-selection limitations persist.

SQ-4 — PROBABILITY-BASED OR HYBRID

Known election probabilities or strong hybrid design exist.

SQ-5 — REPLICATED AND CALIBRATED

The design has been replicated in different waves and external applications; bias and variance behaviour have been calibrated. These statuses may affect the confidence class of the public score.

50. SYNTHETIC END-TO-END GEO-1000 ALLOCATION

SYNTHETIC METHODOLOGY DEMONSTRATION / The following countries, populations, languages, AI product, and calculations are entirely fictional. For the Synthetic Orion AI product, let the suitable population for Native Reach be 1 billion people. Main Population Panel objective:

n=1,000

50.1. Country and Small Country Pool Allocation

50.2. Language Allocation in Country B

Eligible population in Country B:

• Language X: 80 per cent
• Language Y: 20 per cent

Population Panel allocation:

• Language X: 200
• Language Y: 50

Language Fairness additional allocation: Language Y: 50 Field target in Country B:

• Language X: 200
• Language Y: 100
• Total: 300

However, global Population Panel population weights:

• Language X: 80% of Country B
• Language Y: 20% of Country B

remain as such.

50.3. Country Observer Appendix

There are two probability observations for H, I, and J in the small country pool in the Population Panel. The Country Observer Panel also:

H: 1

I: 1

J: 1

can target observations. These three observations:

• increase the geographical coverage,
• show individual country events,

do not create a country score.

50.4. Accessibility Supplement

If the main panel is found to inadequately cover accessibility users, 40 additional observations can be taken. These users: are calibrated according to relevant population groups, produce a separate Accessibility Panel result.

50.5. Total Field Volume

The name GEO-1000 is still valid. Because:

1,000 is the valid observation target for the main population panel.

Complementary justice and coverage panels can be added to this.

50.6. Invitation Volume

On the Population Panel on average:

• completion: 82 per cent
• validity: 92 per cent

let it be expected. Required invitation:

⌈1,000 / (0.82 × 0.92)⌉ = 1,326

may be. With complementary panels, total invitations may exceed 1,500. Therefore:

GEO-1000 is not only a method to send invitations to 1,000 people.

50.7. Field Result

On the Population Panel:

• 1,326 people have been invited,
• 1,080 tasks completed,
• 1,012 records have been found valid in terms of protocol,
• the first 1,000 were included in the main analysis according to a predefined random/backup order

Alright. The remaining 12 valid records:

• backup verification,
• sensitivity analysis,
• recheck

It can be stored for. It cannot be decided which 1,000 will be chosen by looking at the results.

51. NORMATIVE ALLOCATION ALGORITHM

The default sequence of operations for a GEO-1000 main panel is as follows:

Step 1 — Lock the Product and Wave ID

Determine which AI product, surface, and date are measured.

Step 2 — Lock the Target Population Version

The product-eligible population frame from section 5 is used.

Step 3 — Define the Main Current Observation Target

Default candidate target:

n=1,000

Step 4 — Define Material Layers

Country, language, and if necessary, plan/interface layers are defined.

Step 5 — Set Up Small Country Pools

Countries that will generate near-zero allocations are included in appropriate pools.

Step 6 — Calculate Raw Proportional Quotas

qh=nWh

Step 7 — Perform Integer Allocation

Largest Remainder or pre-declared equivalent method is applied.

Step 8 — Add Oversampling Plan

Country Observer, Language Fairness, and Accessibility supplements are saved separately.

Step 9 — Calculate Invitation and Backup Numbers

Expected completion and validity rates are used.

Step 10 — Randomise Selection Order

Primary and backup lists are frozen.

Step 11 — Save Weight Formula

Design, response, and calibration weights are defined.

Step 12 — Confirm Sample Lock

A version and integrity record is created before AI responses are viewed.

52. MANDATORY NORMATIVE PROVISIONS

CH06-N01

The GEO-1000 sample allocation must be created and versioned before AI responses are viewed.

CH06-N02

The main Population Panel target should be defined as at least 1,000 valid observations per AI product and main measurement wave; if a different target is used, the rationale must be explained.

CH06-N03

The invited person, the person completing the task, and the valid observation numbers must be distinguished from each other.

CH06-N04

The main allocation should be based on the product-eligible population defined in Section 5.

CH06-N05

Country or language quotas should be calculated based on the share of the relevant product-eligible population, not the total population.

CH06-N06

The method for converting raw fractional quotas into whole numbers must be defined in advance.

CH06-N07

The resolution of rounding ties must be independent of the results.

CH06-N08

Small countries with a positive target population should not be left as separate zero-probability deterministic strata; an appropriate pooling or probabilistic selection method should be used.

CH06-N09

Small country pools should be justified in terms of product access and financial population characteristics.

CH06-N10

Country Observer observations should also be recorded without changing the main population weight.

CH06-N11

A Country Observer observation cannot be presented as a country score.

CH06-N12

Over-sampled countries, languages, or user groups should be returned to the true target population weights in global estimation.

CH06-N13

The sample proportion cannot be used like the population proportion.

CH06-N14

The same person cannot be duplicated in the global human population due to multiple languages, accounts, sessions, or products.

CH06-N15

Invalid observations should be classified without basing the reason for invalidity on the content of the results.

CH06-N16

Incorrect, negative, rejection, or Critical AI responses cannot be the reason for changing a participant.

CH06-N17

Backup participant lists must be created within the stratum before the results are seen.

CH06-N18

Layer gaps cannot be quietly closed by another country or language.

CH06-N19

Post-result quota, layer, small country pool, or weight changes require a new analysis version.

CH06-N20

All components of the weights and the method of normalisation must be published.

CH06-N21

The assumptions of the nonresponse adjustment must be explained.

CH06-N22

If weight trimming is used, the untrimmed result and sensitivity analysis must be retained.

CH06-N23

The main report should show the appropriate effective sample size alongside the raw number of observations.

CH06-N24

Clustering, repetition, and paired user dependencies should be taken into account in the calculation of variance.

CH06-N25

The design in which the same person tests multiple AI products cannot be presented as a completely independent user sample.

CH06-N26

Each AI product must separately reach its own target of 1,000 valid product observations.

CH06-N27

Allocations Native Reach and Common Support must hold separate sample records.

CH06-N28

Current and fixed-frame results among the waves cannot be combined without explanation.

CH06-N29

If the sample is insufficient for a subgroup, the correct status should be INSUFFICIENT SAMPLE, EXPLORATORY, or NOT ESTIMATED.

CH06-N30

The raw sample size cannot be used as a definitive precision indicator, ignoring weights and design effect.

CH06-N31

Design weights cannot be claimed in a volunteer or non-probability panel as if selection probabilities are known.

CH06-N32

The sample cannot be modified post hoc to exclude low-scoring groups or to enlarge high-scoring groups.

CH06-N33

Observations from the main Population Panel and complementary panels should be labelled separately in terms of purpose and weight.

CH06-N34

The sample lock must have an accountable human or institutional owner.

53. FORMS OF FAILURE

CH06-F01 — SET QUOTA AFTER RESULT

The number of countries and languages is determined after viewing AI responses.

CH06-F02 — USE BACKUP UNTIL BEST ANSWER

Negative responses are replaced with a new user even if valid.

CH06-F03 — COUNT 1,000 INVITATIONS AS 1,000 VALID OBSERVATIONS

Non-completion and invalidity are hidden.

CH06-F04 — RESET COUNTRY TO ZERO WITH PURE ROUNDING

The selection probability of a country with a positive target population is left at zero.

CH06-F05 — GIVING ONE PERSON TO EACH COUNTRY AND CALLING IT THE POPULATION RATIO

The disproportion caused by the minimum quota is hidden.

CH06-F06 — CONSIDERING Country Observer observation AS POPULATION OBSERVATION

The additional observation of a small country receives full weight in the main panel of 1,000 people.

CH06-F07 — ESTABLISHING THE SMALL COUNTRY POOL BASED ON RESULTS

A country producing low scores is moved to another pool or excluded from coverage.

CH06-F08 — PUTTING INCOMPATIBLE COUNTRIES INTO THE SAME POOL

Countries that differ in product access, language, or jurisdiction are combined solely because they are small.

CH06-F09 — OVERWEIGHTING EXCESSIVE SAMPLING

Determines the global result with a sample share higher than the language or country population share.

CH06-F10 — MULTIPLYING MULTILINGUAL INDIVIDUALS

A person is counted as a separate population unit in each language cell.

CH06-F11 — CONSIDERING SAMPLE SHARE AS POPULATION SHARE

Raw response numbers are converted directly into the global score.

CH06-F12 — ARRANGING INVITATION ORDER BY EASE ORDER

Users who respond first and quickly determine the panel.

CH06-F13 — USING STRATUM-OUT RESERVES

Missing country or language observations are quietly completed with participants from another group.

CH06-F14 — LINKING INVALIDITY TO RESPONSE CONTENT

Incorrect AI response is technically declared invalid.

CH06-F15 — REPLACING REFUSALS WITH A NEW USER

System refusal is removed from the denominator even though it is the actual user outcome.

CH06-F16 — HIDING THE WEIGHT FORMULA

The public score does not show with which population and sample weights it was produced.

CH06-F17 — TRIMMING WEIGHTS TO DELETE SMALL GROUPS

Small groups with high weights are trimmed because the result does not look good.

CH06-F18 — HIDING THE EFFECTIVE SAMPLE

Even if 1,000 raw records drop to 500–600 data points due to weights, it is presented as “1,000-person precision”.

CH06-F19 — CONSIDERING CLUSTERING AS INDEPENDENCE

Observations from the same panel, institution, or user are considered completely independent.

CH06-F20 — CONSIDERING THE SAME 1,000 PEOPLE AS TEN THOUSAND INDEPENDENT PEOPLE

Ten observations of an AI product are described as if the population were ten times larger.

CH06-F21 — CONSIDERING COMMON SUPPORT AS NATIVE REACH

The common comparison sample is presented as if it were the product’s actual global reach population.

CH06-F22 — DIRECTLY RANKING NATIVE REACH RESULTS

Product scores in different target populations are compared as if they were from the same universe.

CH06-F23 — HIDING POPULATION CHANGE IN WAVES

Even though new countries or user groups enter the score, the trend is interpreted only as a change in product quality.

CH06-F24 — PRODUCING DEFINITE SCORES FROM INSUFFICIENT SAMPLES FOR A SUBGROUP

A few dozen observations are published as definite scores for a country or language.

CH06-F25 — SILENTLY CHANGING THE SAMPLING LOCK

Quota, weight, or pool definitions are updated without a version record.

CH06-F26 — BLINDLY MULTIPLYING ELIGIBILITY RATES

Eligible population and quota calculations ignore the characteristics of the dependent population.

CH06-F27 — REPLACING THE MAIN PANEL WITH THE COMPLEMENTARY PANEL

Language fairness or Country Observer observations are used to reduce the main population target of 1,000.

CH06-F28 — DELETING LOW-SCORED CELLS WHEN THE PANEL IS SHORT

Incomplete samples are excluded because they are disadvantageous in terms of results.

CH06-F29 — USING WEIGHTING AS SCIENTIFIC FLAIR

It is claimed that the representation problem is solved only by adding weight without explaining selection and coverage bias.

CH06-F30 — THINKING THAT GEO-1000 SHOULD BE ASKED TO EXACTLY 1,000 PEOPLE

Invitation, complementary panels, invalidity, repetition, and effective sample structure are ignored.

54. AUDIT PROCEDURE

Step 1 — Record the Sample Purpose

Global population estimate is divided into sub-group diagnosis, country scope, language fairness, accessibility, and product comparison purposes.

Step 2 — Validate Target Population Version

The population file and date record in Section 5 are reviewed.

Step 3 — Define the Main Current Observation Target

The default GEO-1000 target or a justified alternative is written.

Step 4 — Establish Strata

Differences in country, language, and material user surface are determined.

Step 5 — Identify Small Country Pools

Each pool's:

• countries,
• population,
• justification for similarity,
• internal selection method

is recorded.

Step 6 — Calculate Raw Quotas

qh=nWh

The formula or a justified alternative is used.

Step 7 — Apply Integer Assignment

Rounding and tie-breaking rules are recorded.

Step 8 — Add Complementary Oversampling

Country Observer, Language Fairness, and Accessibility goals are defined separately.

Step 9 — Calculate Invite and Reserve Volume

Completion and validity expectations are used.

Step 10 — Freeze Selection and Reserve Order

Participants are selected or ordered before the results are seen.

Step 11 — Calculate Design Weights

Selection probabilities and layer weights are recorded.

Step 12 — Define the Procedure for Staying Silent and Not Responding

A previously announced response is written to the layer gap.

Step 13 — Create the Sample Lock

Document ID, version, date, and integrity record are added.

Step 14 — Review the Post-Field Validity Flow

It is verified that validity decisions are given independently of the accuracy of the responses.

Step 15 — Generate the Final Weights

The design is separated into response and calibration components.

Step 16 — Calculate the Effective Sample Size

The difference between the raw number and the volume of information is reported.

Step 17 — Compare Allocation and Actuals

Planned and actual observation numbers are shown on a cell-by-cell basis.

Step 18 — Review Post-Result Changes

Quota, pool, weight, and scope changes are checked.

55. REQUIRED EVIDENCE

AI-product identity; measurement wave; target-population version; Population Panel target; Country Observer target; Language Fairness target; Accessibility Panel target; stratum list; stratum populations; population weights; raw quotas; rounding method; fractional-allocation and remainder records; tie-breaking rule; small-country pools; within-pool selection probabilities; country and language allocations; plan and interface allocations; oversampling justifications; invitation targets; completion assumptions; validity assumptions; primary and reserve participant lists; selection or randomisation record; nonresponse records; reasons for invalid observations; validity-decision dates; design weights; nonresponse adjustments; calibration coefficients; weight-trimming policy; raw and effective sample sizes; clustering and matched-user records; Native Reach allocation; and Common Support allocation.

Fixed and current frame weights Comparison of planned and actual allocation Sample lock Change log Responsible person or institution

56. AUDIT CHECKLIST

Is the main valid observation target open? Is the target 1,000 invites or valid observations? Was the sample frozen before AI responses were seen? Which population frame was used? On what basis were the strata established? Are raw country and language quotas shown? Is the method for converting fractions to whole numbers clear? Was the break-off rule predetermined? Is the selection probability for small countries zero? Are small country pools logically consistent? Have Country Observer observations been mixed into the main population weight? Has a country score been produced from a Country Observer observation? If language and accessibility groups were oversampled, were they reweighted? Has the same person been counted as more than one population unit? Are the numbers of invites, completers, and valid observations listed separately?

Have backup lists been created in advance? Has a participant been replaced due to an incorrect or negative AI response? Was the validity decision made before the material accuracy score? Was the tab gap quietly closed from another country? Has the quota or pool changed after the results? Can the design weights be reproduced? Are the assumptions of the nonresponse setting clear? Has weight trimming been done and its effect shown? Are raw and effective sample sizes provided together? Has the same user tested more than one AI product? Has this dependency been preserved in the analysis? Has each AI product reached 1,000 product-observations separately? Were the Native Reach and Common Support samples separated? Did the population frame change between waves?

Have fixed and current framed trends been separated? Have insufficient subgroup samples been published as if they were definitive scores? Is the sample quality status correct? Have complementary panels replaced the main panel? Is there a clear accountable owner of the sample registration?

57. OBJECTIONS AND RESPONSES

Objection 1 — "Why don't we give each country individuals directly according to the population ratio?"

This is a strong assumption in large and medium population countries. In very small countries, the raw quota may fall below one person. In this case:

• small country pool,
• probabilistic country selection,
• separate Observer panel

may be more accurate.

Objection 2 — “Wouldn't giving at least one person to each country be fairer?”

It might be useful in terms of geographic coverage. It is disproportionate in terms of population estimation. The correct solution is to separately label this one person as a Country Observer observation.

Objection 3 — “Doesn't a country receiving zero people on the main panel completely exclude it?”

It can be a problem if a country receives a zero quota as a separate deterministic layer. A positive selection probability can be given in the small country pool. Additionally, the Observer Panel can maintain the observational visibility of the country.

Objection 4 — “Does pooling small countries erase cultural differences?”

It can be deleted if used to generate a country score. For the main global estimate, the pool manages the probability of selection. The country identifier is preserved in the observation record. Separate Observer and country-specific sampling can be done.

Objection 5 — "Why don't we include Observer users within the 1,000?"

They can be included. But then the sample of 1,000 people is no longer proportionate to the pure population. Weighting would be needed, and the effective sample size of the main estimate could decrease. In the founding architecture, 1,000 observations are allocated to the Population Panel; Country Observer observations are added to this.

Objection 6 — "Won't the total field cost become very high?"

It could. The standard should not produce artificial precision by hiding the cost. With a lower budget:

• fewer AI products,
• fewer languages,
• smaller pilot,
• wider uncertainty

can be used. The correct status should be clearly written.

Objection 7 — “Why are we collecting 1,093 or more observations when the name GEO-1000 exists?”

1,000 is the valid observation target for the main population panel. Justice and coverage panels can be added to the main target. This does not compromise the name of the standard. It makes the actual total field volume of the measurement visible.

Objection 8 — “Why would oversampling a language distort the global result?”

An oversampled language appears large because it is in raw responses. When returned to the population weight, the global result is not distorted. The separate language result becomes more precise.

Objection 9 — 'The weights are very complex; isn't the raw average easier to understand?'

The raw mean can be meaningful if it is proportional to the sample population. Excessive sampling, non-response, or coverage differences can cause the raw mean to answer the wrong question. Complexity cannot be hidden if it arises from the sample design.

Objection 10 — “If there are 1,000 valid observations, why should the effective sample be smaller?”

Because some observations can carry very high weight, while others can carry very low weight. Records from the same panel or user may resemble each other. The raw number is the count of physical records. The effective sample approaches the volume of statistical information.

Objection 11 — “Wouldn't it be fairer for the same people to test all AI products?”

Reduces user difference. However:

• it can narrow the product access universe,
• it can increase task load,

it can create contamination between responses. It can be managed with matched or balanced incomplete block design.

Objection 12 — “If a layer does not reach the target, can the global score never be published?”

Not in every case. The deficiency is evaluated in terms of:

• population weight,
• risk,
• the reason for inadequacy,
• remaining coverage

considered. The score:

• with narrowed coverage,
• with wide uncertainty,
• With the warning INSUFFICIENT COVERAGE

it can be published. The deficiency cannot be hidden.

Appeal 13 — “If population data changes, do we have to renew all quotas in every wave?”

Renewal may be required for the current instantaneous score. Fixed starting weights can also be maintained for trend comparison. The two results should be published separately.

Appeal 14 — “If we cannot establish a probabilistic world panel, can NOMOS not be applied?”

It is applicable. However, the sample quality status must be correctly recorded. A panel of volunteers calibrated to the population can be valuable. It cannot be presented as a fully probabilistic sample.

COMMON PROVISION OF CHAPTER 60

The power of GEO-1000 does not arise solely from the number 1,000. Of 1,000 people:

• from which population they came from,
• with what probability they were selected,
• which country and language they represented,
• how many times the same person was counted,
• which observations were considered valid,
• how small countries were managed,
• how over-sampled groups were weighted,
• how many people did not complete the task,
• who was replaced based on the results

A sample's authority arises from these design choices. A defective sample may produce 100,000 responses and still speak about the wrong population. A sound sample may contain fewer responses yet support a more reliable conclusion within clearly stated limits. A large country's population should carry substantial weight in the global human experience, but users must not be fabricated where the product is unavailable. A small country's global weight should remain small, but the country need not be invisible. A language may account for a small share of the global population while still carrying a severe, measurable error. One person may use three languages and five products; that does not create fifteen people. An AI response may be wrong without making the participant invalid. The wrong response is what the audit is trying to measure.

Replacing it with a new user does not clean the measurement. It erases reality. Weighting does not miraculously turn the wrong panel into the world population. However, properly designed oversampling can make subgroups visible without distorting the global estimate. Therefore, NOMOS's sixth measurement law is:

A sample should be established not to produce the result, but to observe the result fairly.

Its seventh law is as follows:

The visibility of a small country does not have to enlarge its population weight.

Its eighth law is as follows:

Excess observation does not mean excess population weight.

The ninth law is as follows:

A wrong AI answer is not data defect; if observed under the valid protocol, it is a matter of oversight.

The tenth law is:

The confidence in a thousand observations does not come from the number thousand; it comes from the reproducibility of the selection and weighting.

Order of Chapter 6 of NOMOS

Don't just tell me, 'We asked 1,000 people.'

Show from which billion people these 1,000 were selected and with what probability.

Don't ignore a country just because it accounts for half a person. / But also, don't claim you measured the whole country just because you sampled one person.

Select small countries probabilistically in the appropriate pool. / If visibility is needed, set up a separate observer panel.

You can oversample low-resource languages. / But do not make it larger in the global score than its population.

Do not multiply a person as much as accounts, languages, sessions, and AI products.

Use the backup when the participant does not respond. / Do not change your user when AI gives a wrong answer.

Decide on validity without seeing the correctness of your answer.

If the quota is missing, explain it. / Do not bring a person from another country to cover the gap.

Do not change the share, weight, country pool, or sample target after seeing the result.

Show the raw number. / Show the effective sample. / Show who received heavy and who received light weight.

Lock the population first. / Then set up the layers. / Then calculate the quotas. / Then select the backups. / Then write the weights. / And only after this, send the first prompt.

The Chapter's Closing Sentence

GEO-1000 is not a method of compiling a thousand answers; it is a protocol for making the representation right of every person in the target population measurable under the probability of selection and correct weighting.

Normative Core

For each AI product and principal measurement wave, the GEO-1000 Population Panel SHOULD target at least 1,000 valid product-observations unless a different target is explicitly justified. Sample allocation MUST be designed, versioned, and locked before AI responses are observed. Primary allocation MUST be based on the product-eligible target population, not raw resident population, invitation counts, accounts, sessions, or convenience availability. Fractional allocations MUST be converted to integer quotas using a predeclared, reproducible, outcome-blind method. Positive-mass populations MUST NOT be deterministically assigned zero selection probability when the audit claims inference to the full target population. Small populations MAY be represented through justified pooled strata and probability-proportional selection. Country Observer, Language Fairness, and Accessibility observations MAY supplement the main panel, but MUST remain separately labelled and appropriately reweighted. Oversampled groups MUST NOT receive their raw sample share as population weight. Participant replacement MUST be based only on predeclared recruitment or protocol-validity conditions. Incorrect, negative, refused, or critical AI outputs MUST NOT trigger replacement. Persons, accounts, sessions, languages, and AI-product observations MUST remain distinct units. Design weights, nonresponse adjustments, calibration, trimming, clustering, paired observations, and effective sample size MUST be documented. Any post-result change to quotas, strata, pools, weights, exclusions, or sample targets MUST create a new analysis version and preserve the original record.