Chapter VIII — Audit

From Claim to Decision

An audit does not manufacture validation. It decides the limits within which a claim may speak. It is not another ring, but the decision mechanism through which the preceding rings work together. The Centre limited the object; Evidence the sentence; Measurement the observation; Intervention the action; Governance the authority; Time the validity; and the Final Test the value to the business. Audit unites them in one traceable file.

A business may purchase an audit service; it cannot purchase a favourable outcome. A practitioner may receive a fee; they cannot sell approval. “We are strong in AI systems—prove it” is a publication request whose result has already been written. “Assess how generative systems represent us, how that representation relates to customer outcomes, and which statements we may make” is an auditable request.

An audit does not begin by collecting data. It begins by opening the sentence to be adjudicated. What exactly will be said? Who will use it? Which decision will rely on it? What are its scope and risk? Until when will it remain valid? The audit asks one further question: which adverse outcomes is the client unwilling to accept in advance? If the business permits only a favourable judgment, the audit does not begin. A process whose outcome is not genuinely open is not an audit. It is a staging.

1. The Seven Questions of Audit

Every audit answers seven questions:

1. Which entity and which representation are under examination?
2. Which evidence about the real world, representation and outcomes supports the claims?
3. Under which protocol were the observations measured?
4. If distortion exists, is intervention necessary, proportionate and legitimate?
5. Who made the decision, and with whom did accountability remain?
6. Are the records and authorities still valid?
7. Where the scope is commercial, did AI influence leave sustainable value?

An audit need not address every question at the same depth. Identity representation alone may be examined; the work is not invalid, but it cannot produce a commercial judgment. Representation in specified systems and languages may be assessed; a mark of conformity may not follow. An omitted field is not a defect. A concealed omission is.

The output is not one score but an Audit Opinion. It may state that a claim is unsupported; can be carried only within a limited scope; that a material distortion was verified; no grounds for intervention were found; a limited intervention is appropriate; commercial contribution was observed; sustainable contribution was not demonstrated; the outcome is invalid because of ethics or governance; or the records have expired. The value of an audit does not lie in producing a favourable result. It lies in its ability to carry an adverse one.

2. The Audit Admission Gate

Not every request is admitted to audit. Six conditions apply.

Entity clarity: The company, brand, person, product, service or group is defined precisely. A holding company and one of its brands, or an author's legal identity and pen name, may be related without constituting the same audit unit. If the object is ambiguous, work does not begin until scope is clear.

Decision purpose: There must be a definite decision question, such as verifying a representational distortion, determining the need for intervention, evaluating a public claim, examining the relationship with customer outcomes, testing sustainable contribution or reopening the validity of an earlier decision. “Let's see how we are doing” is not enough; if the decision is unknown, sufficient evidence cannot be defined.

Access to records: Reasonable access is provided, as the scope requires, to formal identity, product and service boundaries, licences, site versions, previous measurements, intervention history, CRM, sales, delivery, refunds and customer accounts. Inaccessible records are not assumed away; they narrow the scope. A representational review may proceed without commercial records; a sustainability audit may not.

Acceptance of an adverse outcome: The parties accept that the claim may be rejected, public language narrowed, intervention not recommended, commercial effect left unproved, conformity refused or an earlier decision withdrawn. A party that contractually prohibits an adverse finding cannot be audited.

Clarity of conflicts of interest: Relationships among auditor, practitioner, entity owner and any prospective conformity programme are recorded. Outcome-based fees, services to a competitor, the auditor's own implementation of the intervention, or use of a favourable report in an investment decision or contract renewal are disclosed. One institution may implement and audit an intervention, but in a high-risk case it may not approve its own result without an independent second decision.

Synthetic data, confidentiality and privacy: Synthetic, anonymised and composite records are labelled explicitly. Trade secrets and personal data are protected, but protection does not create a right to conceal an omission that would alter the judgment. Details may remain private; the limitation must be public.

The gate may decide admit, conditional admission, admit within limited scope, or refuse. Refusal may mean lost revenue. It is not a loss of the standard.

3. Scope Lock and Audit Levels

Before the audit begins, the entity, geography, language, system and interface, time, representational field, prompt universe, comparison group, commercial field, risk, audit level, public use, validity and exclusions are locked. Scope states not only where the audit looks, but the boundary the judgment may not cross.

A new system, language or record may be added during the work, but never by silent expansion. A new element creates a new scope version. An English-language result is not evidence of German representation; a brand audit is not evidence about the group of companies; identity conformity is not evidence of commercial sustainability.

There are five audit levels:

1. Preliminary Review: Determines whether an auditable problem, claim, initial risk and appropriate scope exist. It produces no public statement of success.
2. Representation Audit: Examines evidence about the real world, a controlled prompt universe, contrary and exclusion tests, human coding and the representation profile. It may speak about material distortion; it may not speak about commercial contribution.
3. Intervention Audit: Tests before-and-after change through a frozen baseline, the Intervention Gate, intended and prohibited outcomes, core set, observation window and withdrawal conditions. It may show change; it does not guarantee full causation.
4. Commercial and Sustainable Value Audit: Combines influence class, customer cohort, qualified demand, sales, net contribution, delivery, satisfaction, repeat behaviour, channel dependency and an adequate observation horizon.
5. Conformity Assessment: Examines business and practitioner fitness within a defined scope of the Framework, evidence and measurement discipline, intervention governance, public language, time, conflicts of interest, and suspension and withdrawal conditions. Completing this level does not itself create a badge. Without the independent programme infrastructure required by Chapter V, it produces only a pilot conformity opinion.

A lower level may not borrow the language of a higher one.

4. The Audit File and Claim Register

An audit is not one report but a system of linked records. The Entity File carries the entity under audit and its relationships; the Claim Register every sentence the parties seek to make; the Evidence Record the support for each sentence; the Measurement Record the observations; the Intervention Record the change and withdrawal conditions; the Governance Record authority, conflicts of interest and accountability; the Time Record validity and version; and the Sustainable Value Record the business outcome. Approaches such as W3C PROV-O, which express provenance through relationships among entities, activities and agents, offer a technical neighbour to this idea of traceability; the Framework does not require the use of a particular ontology.[9]

No record substitutes for another. Measurement shows the observation; the Evidence Record permission for a sentence; Intervention the operation; Governance authority for the operation; Time its current validity; and the Value Record the business's gains and losses. The opinion is not constructed afterwards to justify these records. It arises from them. Documentation approaches such as model cards and datasheets for datasets have shown, on different planes, the importance of recording purpose, scope, performance, composition, use and limitations.[12][13] Other approaches to AI audit likewise treat documented roles, evidence and corrective mechanisms across the lifecycle as essential to accountability.[10] NobleJackal's file architecture is adjacent to this literature; it has not been validated by it.

The Claim Register is especially important. “Systems recognise us”, “We are represented with accurate attributes”, “We are recommended more often than competitors”, “The intervention increased visibility”, “AI brings us customers”, “These customers are more profitable” and “We conform to the Framework” are not the same sentence. Each carries a separate type, owner, intended use, risk, required audit level, linked record, permitted wording, prohibited extension, period of validity and approval.

A material sentence not entered in the Register cannot be made public. The claim “AI recommends us” may be rejected while the sentence “The brand was recommended in six of the specified prompts” is permitted. The claim has become smaller and moved closer to the truth.

5. The Audit Protocol

An audit proceeds in ten steps:

1. Freeze the request: Write down the decision sought and the proposed public sentence; a change opens a new version.
2. Lock entity and scope: Define systems, languages, time, market, prompt universe and excluded fields.
3. Create the Claim Register: Separate claims about the real world, representation, intervention, commercial effect and conformity.
4. Establish the real-world baseline: Verify identity, activity, capacity, price, service boundaries, licences and the reality of delivery.
5. Lock the measurement protocol: Define the core and exploratory sets, repetitions, codes, evaluators, denominator, missing data and time in advance.
6. Produce observations of representation: Test supportive, neutral, contrary and exclusion fields together; preserve adverse outputs as well.
7. Code and adjudicate: Assess entity, attribute, judgment, source support, currency and the relevant commercial trace separately.
8. Establish the commercial connection: Where in scope, examine influence classes, cohorts, sales, cost, delivery and repeat behaviour.
9. Decide on intervention or conformity: Apply the Intervention Gate where distortion exists; postpone or refuse a decision where evidence is inadequate.
10. Publish the Audit Opinion: Present the decision together with scope, limitations, validity, conflicts of interest and review.

The outcome is written in the final step. It is not selected in the first.

6. Sampling, Sufficiency and Human Judgment

The Framework does not prescribe one universal number of prompts for every audit. Thirty prompts are not sufficient for every business; three hundred are not necessary for every business. Sufficiency is justified in relation to audit level, variation in representation, number of systems and languages, prompt families, risk, user effect, consequence of the decision, frequency of contrary results and temporal distribution.

Sample size cannot change after the outcome is seen. Testing may not stop because a favourable pattern appeared early, nor may the prompt universe be enlarged arbitrarily because an adverse result emerged. The stopping rule is written in advance. Observational sufficiency may require, together, completion of the predefined sample, testing of contrary and exclusion fields, and the point at which new records cease to produce a material change in the profile. Sufficiency is not universality: forty prompts carry a judgment only for their set, and two languages only for those languages.

Automation may assist with grouping, similarity, missing fields and preliminary coding; it cannot independently validate its own output. High-risk work requires at least two human roles. Where evaluators disagree, each records their rationale, the common definition is applied again and a third person decides if necessary. The initial disagreement is not erased. Agreement between two people does not create truth, but it strengthens the consistency of judgment.

7. Finding Profile and the Prohibition on a Single Score

An audit does not allow strength in one field to compensate for weakness in another. Its finding profile therefore retains separate cells:

Table 8.1 — Audit finding profile

An ethical breach vetoes the positive judgment; invalid measurement does not permit a strong representational claim; sustainable commercial contribution does not make a distortion in representation true. One score hides contradiction. The profile makes it visible.

8. Public Summary, Confidential Annex and Re-audit

An audit may produce two outputs. The Public Audit Summary states the entity, purpose, scope, level, Framework and protocol versions, systems and languages, time range, principal findings, permitted wording, exclusions, conflicts of interest, use of synthetic or anonymised data, validity and review.

The Confidential Audit Annex may protect complete prompts, raw responses, customer records, commercial calculations, personal data, contracts, security information and detailed evaluator notes. Confidentiality does not create a right to mislead in the public judgment. The number behind a high refund rate accompanying positive contribution may remain private; the material limitation may not. Commercial details of a conflict may be withheld; the conflict's existence is disclosed.

Re-audit opens when the company, brand, product, country, language, service or pricing model changes; a material system or protocol change occurs; strong counter-evidence or serious customer harm appears; an unexpected side effect, scope breach or new conflict of interest emerges; the validity period ends; a foundational release is made; or commercial outcomes reverse. The earlier opinion is not erased; it is assigned a status of active, conditional, under review, superseded, expired, withdrawn or archived.

9. Teaching Case: Kestrel Grid Services

Teaching simulation — synthetic data. This case does not represent a real energy company, customer dataset or conformity programme.

Kestrel Grid Services is a fictional provider of electrical-grid maintenance and fault-prevention services for industrial facilities. A Level IV audit covers twelve months and two complete delivery cycles.

Table 8.2 — Synthetic audit outcome for Kestrel Grid Services

The records do not show that AI influence was the sole cause. They do show that the associated cohort left positive contribution, value was delivered and repeat behaviour emerged across two business cycles. There is no comparative evidence for a superiority or “best company” claim.

The Audit Opinion is therefore limited as follows: “During the specified twelve-month period, six projects in the cohort associated with AI influence through multiple records were completed profitably; three repeat contracts and two referral enquiries were observed. No claim is made that AI influence was the sole cause or that Kestrel is superior to its competitors.”

The case may receive a favourable finding in a design test of a conformity programme; no badge is issued because an active, independent programme has not been established. A favourable Audit Opinion gives the business not every sentence it wants, but the sentence its evidence can carry.

10. The Audit Opinion

An opinion is not a free-form paragraph of celebration. It states the entity under audit, level, scope, Framework and protocol versions, time, principal findings, decision, permitted wording, prohibited extensions, exclusions, conflicts of interest, validity and review.

A limited representation opinion might read: “In the Representation Audit conducted across the specified systems, languages and dates, entity identity was assessed as mostly consistent, with recurring omissions in two core attributes. Commercial effect was outside scope.” A refusal might read: “The submitted records do not support the claim ‘Systems recommend the business as an industry leader’; public use has not been permitted.”

A sound Audit Opinion does not hide uncertainty. It states which judgment that uncertainty prevents. It need not appear absolute. Its boundary must be.

The judgment of this chapter is:

Audit does not validate an outcome; it tests a claim. Where evidence is sufficient, it permits the sentence; where evidence is insufficient, it narrows the sentence; where the boundary is breached, it rejects the sentence.

The business is not the only subject under audit. The auditor's independence, the standard's integrity and the opinion's own boundary are audited too. A fee may start the process; it cannot determine the judgment. A favourable finding may be recorded; it cannot guarantee the future. A decision that cannot be withdrawn is not authority but obstinacy.

A standard is not complete merely because it explains how to audit. It must also carry the strongest objections that can be made against it.