AI PROVIDER COST TRANSPARENCY INDEX — METHODOLOGY

Status: METHODOLOGY FROZEN 2026-08-27 · v1.0 (was v0.1, RFC open 2026-08-12 → 2026-08-26) Owner: Llorenç "Nzo" Ballester Target: part of the AI Cost Transparency Report — Q4 2026 Rule (non-negotiable): NO scoring is published until the methodology is frozen. No fabricated scores, ever. This document is the process — the proof of rigor.

What freeze means, precisely: the rubric (§3), scoring model (§2) and the five open questions (§7) are now fixed for this edition. Per §5 step 2, any change from here resets the whole edition to step 1 — a new RFC, not a patch. Freeze is not scoring: data/transparency/scores.ts stays empty (enforced by tests/transparency.test.ts) until each of the ~10 providers has been evaluated against every criterion with dated primary evidence. That is the next phase, tracked separately — it did not start today.

1. What this Index is (and is not)

• Is: a scored, evidence-based evaluation of ~10 AI providers on cost transparency — how easily a customer can know what they will be charged and see where the money went.
• Is NOT: a ranking of model quality, performance, or value; a sustainability score; an opinion piece. A provider can score well on transparency and still be expensive — the two axes are unrelated.
• Evidence class: EVIDENCE-BASED / CALCULATED (see /methodology). Each criterion is scored from primary evidence (official pricing pages, docs, API output, invoices) captured at a dated snapshot.

2. Scoring model

• Binary per criterion: YES (1) / NO (0).
• NV (not verifiable): no primary evidence obtainable from the public internet at the snapshot date. Treated as 0 in the aggregate but reported separately so the reader sees what could not be verified — the score's own transparency matters as much as the score.
• N/A: criterion does not apply to the provider's offering (only allowed where the rubric explicitly marks it; must be justified).
• Blocks are summed (raw counts), each block reported separately. A single headline number is NOT produced at this stage — block-level scores are the artifact. (Weighting is deliberately deferred to feedback; see §7.)
• No CO2 estimation, no environmental score. Environmental disclosure is scored only as disclosure: does the provider publish clear information, yes or no. We never estimate emissions or grade sustainability — that requires a methodology we do not have.

3. Rubric (draft)

Block A — Pricing clarity

Block B — Billing predictability

Block C — Spend observability

Block D — Billing / invoice quality

Block E — Disclosure

4. Evidence rules

• Every criterion scored YES requires a citation to primary evidence: URL + the exact text/screenshot captured + snapshot date. Stored in the dataset (see §6).
• Evidence is captured at a dated snapshot, never remembered.
• A criterion with ambiguous evidence is NV, not YES.
• One reviewer + one verifier; disagreements resolve to the stricter reading.

5. Process

1. RFC open (this document) — feedback period 14 days. Changes from feedback are recorded in a changelog inside this document.
2. Methodology freeze — the rubric and scoring model become immutable for the edition. Any change after freeze resets the whole edition to step 1.
3. Scoring — per §2/§4. Dataset fills in data/transparency/.
4. Right of reply — each provider is offered 14 days to respond before publication (contact via published channels; if unreachable, noted).
5. Publication — the report and dataset ship together. Dataset licensed CC-BY 4.0 (with attribution) if legal review clears it; otherwise the dataset stays under the site license.
6. Re-edition — next edition re-captures evidence; scores are never "carried over" silently.

6. Data schema

• data/transparency/provider-meta.ts — provider list + pricing URL (evidence base).
• data/transparency/rubric.ts — the rubric as data (id, block, question).
• data/transparency/scores.ts — EMPTY until methodology freeze. Guarded by a test: if any score is present before freeze, CI fails.

// canonical score row (post-freeze only)
export type ScoreRow = {
  providerId: string;
  criterionId: string;
  value: 1 | 0 | "NV" | "N/A";
  evidenceUrl: string;     // primary evidence
  captured: string;        // YYYY-MM-DD snapshot
  note?: string;
};

7. Open questions — resolved at freeze (2026-08-27)

Zero external feedback arrived in the 14-day window (verified: issue #25 carries only the opening comment). Absent that input, each question is resolved here by the owner, using the most conservative reading available — the one that adds the least unverifiable judgment to an evidence-based instrument — rather than left unresolved past the freeze date. Each resolution is itself a documented editorial decision, re-opened only in a future edition's own RFC, never patched into this one.

1. Weights or raw block counts? → Raw block counts, no weights, edition #1. Inventing a weighting scheme ourselves — with no reader or provider input — would put an unverifiable opinion (how much does "spend observability" matter vs "invoice quality"?) inside a score we're calling evidence-based. The RFC already leaned this way (§2: "a single headline number is NOT produced at this stage"); freeze makes it final for the edition rather than provisional.
2. 5 criteria/block: right load? → Kept at 5×5=25. No evidence surfaced that it's too long or short, and the existing rubric (§3) is already the product of the pre-freeze URL audit (2026-08-13) that reshaped the provider list. Revisit with real edition #1 friction, not in the abstract.
3. NV as 0 or reported separately? → Both, as originally drafted (§2): NV counts as 0 in the aggregate, and is always reported next to the score so a reader sees what's a real 0 (published, and it fails the criterion) vs an NV (nothing to verify). Collapsing the two would hide exactly the kind of ambiguity this Index exists to surface.
4. 14 days for right of reply? → Kept. Same length as the RFC's own feedback window; no basis surfaced to lengthen or shorten it for providers, who have more direct incentive to respond than an anonymous reader had to comment.
5. Marketplaces (OpenRouter/AWS/Azure) scored as intermediaries? → Yes, all three stay in the ~10, scored as intermediaries; the underlying vendor is not credited or blamed for the marketplace's own transparency. provider-meta.ts already carries this for AWS Bedrock and Azure OpenAI (notes, since 2026-08-13); this resolution extends the same logic to OpenRouter, which had been left explicitly open. The distinction that matters is not "marketplace vs vendor" but whether the entity publishes ITS OWN pricing, billing and invoice surface — Bedrock, Azure OpenAI and OpenRouter each do, independently of the model they're routing to, which is what makes them scoreable at all.

Changelog

• 2026-08-12 — v0.1 opened for feedback.
• 2026-08-27 — v1.0, methodology frozen. Zero feedback comments received in the 14-day window (2026-08-12 → 2026-08-26); the five open questions in §7 are resolved by the owner per the reasoning above. Rubric (§3) and scoring model (§2) are now immutable for this edition per §5. Next phase: scoring against primary evidence, tracked as its own issue — not started by this freeze.