Record 01 · Evaluations, traces and measured impact

Measure what your AI systems actually do.

Traces in. Scorers over them. Reports with a digest, release gates that re-check the report, people in the loop, and impact with its denominators. EvalBook measures; the products and people around it decide.

Open the app How it works

CodeBook executes · EvalBook measures · MadeBook authorizes · Bookbag SSO authenticates, grants model access and meters usage.

Identity: Bookbag SSO · the organization is the tenant
Model calls: Through SSO · budgeted · ledgered
Customer code: Runner only · outbound · sandboxed
Reports: Immutable · digested · re-fetched
Price at launch: Free

Evaluation report · schematicShape from the peer protocol · not a measurement

suite: release-readiness · v7
mode: release_gate
subject: repository_change · acme/api
commit: <40 hex> · base <40 hex>
subject_digest: sha256:…
dataset_refs: 2 versions pinned
issued_at: —
valid_until: —

CHK-01unit-tests · executablerequiredplatform_verifiedpass
CHK-02json-schema · deterministicrequiredplatform_verifiedpass
CHK-03answer-grounded · judgerequiredreportedfail
CHK-04dependency-scan · executableoptional—unsupported
CHK-05latency-p95 · deterministicoptionalplatform_verifiedpass

coverage

expected 5 · executed 4 · missing 0

evidence_complete

false · CHK-03 is reported, not verified

result

fail

digest sha256:… · the consumer recomputes it over the canonical body and refuses a mismatch · pass only when every required check passed with complete evidence

01The argument

A measurement is not a decision. Keep them apart.

Coding agents and AI applications now produce more than a team can read. The question is no longer whether to evaluate them, but who holds the instrument, and whether the instrument is allowed to pull the lever.

EvalBook is the instrument. It takes traces from any application or from a CodeBook factory, holds the datasets and the scorers, runs the suites, and issues a report that cannot be edited afterwards: every check’s state, the coverage, whether the evidence is complete, and a digest over the whole. A judge’s opinion is labelled as a judge’s opinion, and calibrated against people.

It does not pull the lever. CodeBook decides whether a run passes its station, after fetching the report and recomputing the digest. MadeBook decides whether a report counts as evidence on a pull request, after doing the same. When EvalBook’s own protected configuration changes, it asks MadeBook first, or applies separation of duties when it stands alone. The report is the record; the decision is somebody else’s, on purpose.

02Legend

Nine words, used precisely.

The manual, the application and the peer protocol use the same nouns. Each one is a table with a digest, a tenant and an audit row; none is a loose concept.

ProjectL-01: The work container. One Bookbag organization owns many; a project links to a CodeBook workspace and a MadeBook workspace.
TraceL-02: A tree of spans sharing a trace id: runs, steps, model calls, tool calls, retrievals, evaluator verdicts.
DatasetL-03: Examples with a schema, frozen into immutable, digested versions with splits and slices.
ScorerL-04: Deterministic, executable in a runner, or a model judge. Versioned; a judge is calibrated against human labels.
SuiteL-05: The scorers, dataset versions, thresholds and decision rule an evaluation runs under, frozen as a version.
ReportL-06: The immutable, digested result of one evaluation of one exact subject. A consumer re-fetches and re-hashes it.
Review queueL-07: Where people annotate, score and adjudicate, and where judge calibration gets its labels.
ProposalL-08: A problem, a hypothesis and a bound, sent to CodeBook as one card with protected paths.
Impact contractL-09: A metric from the dictionary, a baseline, a cohort and a method that is named, never called causal.

03The loop

From a span to a measured outcome, and round again.

Each stage is a component of the application with its own tables, routes, jobs and retention. Work crosses between them only through declared interfaces, so a trace never becomes ground truth by accident and a judge never scores its own evaluation spans.

The loop

Traces in · impact out · people and peers between

08 → 01: what impact observes becomes the next trace, the next candidate example, the next failure cluster.

The loop, stage by stage

04What it measures

Six kinds of reading, each with its source.

Every figure the application shows says where it came from — measured, manual, estimated or unavailable — and what it was divided by. An absent number is a dash.

05In the suite

CodeBook executes. EvalBook measures. MadeBook authorizes.

EvalBook is one of the Bookbag products. It needs none of the others to be useful, and it is strongest beside them: a CodeBook station can require a suite, MadeBook can require a passing report on a pull request, and both learn what EvalBook observed about the work they shipped.

authenticates
Bookbag SSO
Accounts, organizations, roles, model access, the token ledger, files, billing, the peer directory and the peer secret.
executes
CodeBook
Factories run the work. A station can require an EvalBook suite and waits for the report; runs export as traces; outcomes are told to EvalBook; proposals arrive as cards.
As a peer →
measures
EvalBook
Traces, datasets, scorers, suites, reports, review, improvement, impact. It issues reports and asks before it changes protected configuration.
authorizes
MadeBook
Reports become evidence on a pull request only after MadeBook re-fetches and re-hashes them; protected EvalBook changes are decided by its policy and people, never by a model.
As a peer →

CodeBook → EvalBook: evaluate this exact candidate · run and step spans · outcome.observed
EvalBook → CodeBook: evaluation.completed (re-fetch the report) · improvement proposals as cards
MadeBook → EvalBook: reports for this head commit? · evalbook_decision.resolved
EvalBook → MadeBook: evaluation.completed · may I activate this suite version? (decision bound to its digest)

Every call: bearer peer secret · X-Peer-App · X-Peer-Signature = HMAC-SHA256 over the raw body · Idempotency-Key on writes · every event carries the link it travels on · a callback never carries the verdict a consumer acts on.

One account. Your Bookbag account signs you in to every product. Organizations, people and roles are managed once, at SSO; EvalBook reads them live and never manages members.
One peer directory. Products find each other and the shared peer secret through SSO’s app registry. Nothing is configured on EvalBook’s server for a peer.
One outbox. Every event EvalBook owes a peer is signed, keyed and retried; after the last attempt it is marked failed, shown, and retryable by a person — never dropped.

06What it never does

Six refusals, enforced by the application.

These are not policies written in a prompt. Each is a rule in the code, with a test behind it, and the manual names the page where it is described.

N-01
It never authorizes.
A report is a measurement. MadeBook decides, CodeBook decides, a person decides. When EvalBook’s own protected configuration changes, it asks MadeBook for a decision bound to the change’s digest, or applies separation of duties in a standalone project — the author never activates their own change.
N-02
It never runs customer code in the API or worker.
Executable scorers run in a separately installed, outbound-only runner, in a container with no network, a read-only root and a read-only harness. The API and worker claim no job that executes anything of yours.
N-03
It never calls a model provider.
Every judge, pairwise, generation and investigation call goes through Bookbag SSO with a reserved budget and an idempotency key. EvalBook holds no provider key.
N-04
It never shows an absent number as zero.
Every metric carries its source — measured, manual, estimated or unavailable — and its denominators. Unmeasured is “—”.
N-05
It never takes the tenant from a request body.
The organization and project come from the API key, the session or the peer link. A batch naming another tenant is refused; another tenant’s row by id is not found.
N-06
It never edits a version, a report or an audit row.
Dataset, scorer and suite versions and reports are rows that are never updated; a change is a new version with a new digest. The audit log is hash-chained per organization.

The security model The runner sandbox

07In the application

12 areas in the application today. None verified yet.

Built and reachable in the shell. The requirements ledger carries each one as implemented but unverified, or as being verified now, and the manual page for each area says so at the top. “Verified” means verified with evidence by the owner’s bar, not by the agent that built it.

Observe

Traces
In the application · unverified
Spans from the SDKs, the HTTP API, OTLP/HTTP (JSON and protobuf) and CodeBook runs, with durable receipts, batch idempotency, a 1,000-span / 5 MB cap per batch, per-trace sampling and quota backpressure that refuses a batch whole rather than half-storing it.
Ledger: EV-002–009 not yet marked implemented; being verified
Capture policy and redaction
In the application · unverified
Per project: full, redacted or metadata-only capture, applied before anything is stored; path and pattern rules with the changed paths recorded; bodies over 256 KB in the organization’s SSO file store; an external-judge body policy that defaults to forbidden.
Ledger: EV-007 not yet marked implemented
Retention, export and deletion
In the application · unverified
Retention per content class (raw bodies, traces, results, cost records, audit metadata) enforced by daily purgers with receipts; JSONL exports; deletions that need the project’s exact name, remove derived rows and files, leave a tombstone and withdraw earlier exports.
Ledger: EV-009 not yet marked implemented

Measure

Datasets with immutable versions
In the application · unverified
Examples with a schema and correction history, frozen into versions whose digest is computed over the examples; splits with families kept together; slices; exact-digest and near-duplicate leakage scans on every freeze; imports that report every rejected row.
Ledger: EV-010–018 implemented, unverified
Curation and synthetic data
In the application · unverified
Candidates from traces, reviews, proposals and incidents wait as pending and are never auto-labelled ground truth; generated examples and suggested expected outputs are labelled unvalidated until a reviewer approves them; holdout examples are never shown to the generator.
Ledger: EV-015/016 implemented, unverified
Scorer registry
In the application · unverified
Versioned scorers of three kinds — deterministic, executable in a runner, model judge — each with a digest, a score type, a direction, a timeout and what it requires; test cases and test runs per scorer; pairwise judges; activation of a version is a governed action.
Ledger: EV-019–028 not yet marked implemented; being verified
Model judges with calibration
In the application · unverified
Judges call models only through Bookbag SSO, with a budget reserved before the call and the charge deduplicated by idempotency key; rubrics are versioned; calibration against human labels reports Cohen’s kappa or Krippendorff’s alpha with its sample count.
Ledger: EV-026/027 not yet marked implemented; being verified
Executable scorers in an outbound-only runner
In the application · unverified
Tests, builds, lint and scanners run on your machine in a container with no network, a read-only root and a read-only harness, claimed by a separately installed runner that dials out and never listens. The API and worker never execute customer code. No release gate relies on a runner result until the register records a live run.
Ledger: EV-021 not yet marked implemented; runner drivers not yet verified live

Account for it

Impact contracts and source events
In the application · unverified
A versioned metric dictionary; source events from signed webhooks and from CodeBook’s outcome.observed; baselines captured and approved; cohorts; a contract names its method — descriptive before/after, matched observational or randomized — and an association is never called causal.
Ledger: IMP-001–022 not yet marked implemented; being verified
Usage and cost, allocated from SSO
In the application · unverified
Every model call EvalBook makes is a ledger row at Bookbag SSO; EvalBook’s own figures are labelled “allocated from SSO usage” with the ledger reference, missing cost is counted as missing, and spans quoting one ledger reference are counted once.
Ledger: IMP-012/013 not yet marked implemented
Peer links and governed configuration
In the application · unverified
Links to CodeBook and MadeBook workspaces from Bookbag SSO’s peer directory; every call signed over its raw body and keyed for idempotency; protected configuration changes ask MadeBook for a decision bound to the change’s digest and fail closed; a standalone project applies separation of duties.
Ledger: verified against stand-ins; never crossed to a real CodeBook or MadeBook
The capability register
In the application · unverified
Every adapter operation with its status (verified live, verified against a stand-in, implemented unverified, in progress, not built), the documentation it was built against and its evidence. No operation is verified live, and no gate relies on any of them yet.
Ledger: register version 2026-10-15.1

What has and has not been verified, area by area, is on Implementation status.

08Pricing

Free at launch, within a Bookbag organization.

EvalBook is not a billable product yet. It reads your organization’s plan from Bookbag SSO only to show it, and it enforces one limit of its own: the free plan holds three active projects per organization. Archiving a project frees a slot; nothing is deleted by a plan.

No card, no trial clock. Sign in with your Bookbag account. The organization you are a member of is the tenant.
Model calls are the organization’s. Judges, pairwise comparisons, synthetic data and investigations call models through SSO within the organization’s allowance, and every call is a ledger row you can see under Usage & cost.
Paid tiers are not decided. When they are, a plan change will never delete evidence, never turn a deny into an allow, and will leave every historical report readable and exportable.

Open the app Signing in

Price at launch: Free
Tenant: A Bookbag organization
Active projects: 3 per organization on the free plan
Model calls: Through SSO, within the organization’s allowance
Paid tiers: Not yet decided

09Being finished

Being finished. Not available yet.

These areas have a contract and, in most cases, an API in the application, but their pages are not in the shell and the requirements ledger has not moved. They are listed here so you can plan around what the product supports now, and nowhere else on the site.

Not availableBeing built
Suites, experiments and reports
A suite version freezes its scorers, dataset versions, thresholds and decision rule; an evaluation pins every version and the subject digest; the report is immutable, digested, with explicit check states (pass, fail, error, timeout, skipped, unsupported, abstained, inconclusive, missing, stale) and coverage.
Ledger: API routed; pages not in the shell; EV-031–037 not started in the ledger
The contract it is built to →
Not availableBeing built
Release gates for CodeBook and MadeBook
CodeBook asks for an evaluation of an exact candidate and waits; MadeBook asks for reports on a head commit. Both re-fetch the report and recompute its digest; pass only when every required check passed with complete evidence. The callback never carries the verdict they act on.
Ledger: EV-038 CodeBook side verified; EvalBook side being finished
The contract it is built to →
Not availableBeing built
Statistical decision rules
Wilson intervals, paired bootstrap with a recorded seed, exact McNemar, Holm–Bonferroni across slices, non-inferiority with a declared margin, group-sequential monitoring — frozen in the suite version before an experiment starts.
Ledger: EV-033–035 not started in the ledger
The contract it is built to →
Not availableBeing built
Human review, online scoring and alerts
Queues with assignment, annotation, correction and adjudication; inter-rater agreement; online scoring policies that sample completed application traces and never evaluation spans; alert rules over scores and drift.
Ledger: API routed; pages not in the shell; EV-029/030/039/040 not started in the ledger
The contract it is built to →
Not availablePlanned
Failure clusters, investigations and proposals
Failure signals deduplicated into clusters by subject and signature; an investigation assistant that calls models through SSO; proposals with a problem, a hypothesis, a reproduction, a bound and protected paths, dispatched to CodeBook as one card per proposal.
Ledger: component is an empty shell; EV-041–047 not started
The contract it is built to →

Free at launch

Measure first. Decide elsewhere.

Open EvalBook, create a project, send a trace. The manual covers the rest, and says on every page what is and is not verified.

Open the app Read the manual