# Nodus GPU Goodput evidence tools (draft 0.2)

This is the first executable extension of [the public protocol](goodput-v0.2-protocol.md) beyond IdleBench: a standard-library, read-only evidence-bundle reader and scorer for `nodus/serve`, `nodus/train` and `nodus/kernels`. It neither launches GPUs nor runs workloads. IdleBench v0.1 and its existing results remain unchanged. Fleet and elastic-cohort scoring are not implemented.

## Produce and score a bundle

A runner writes `run.json` and preserves raw output. Only score after every allocation in the fixed cohort has confirmed release:

```sh
python3 -I goodput-v0.2-score.py /path/to/bundle
```

Exit 2 means malformed, incomplete or ineligible evidence; it does not mean the experiment lost. Failed runs stay in the campaign evidence and publication inventory. A valid score is **not** independent GPU attestation, proof of novelty, a quality audit or product availability. The reader trusts the supplied correctness verdict only after verifying its referenced bytes; independent review must assess that verdict.

All times are seconds in the same recorded clock domain. Record clock synchronization in the configuration for multi-host work. All allocations must contain the measurement window. For a fixed multi-host cohort, each allocation has its own start/release, count and unique ID; GPU-hours are the sum of each count times its entire allocation duration. Overlapping/elastic cohorts and mixed GPU models need a later protocol. The complete manifest hash is emitted with every score, so changed SLOs or metadata cannot masquerade as an unchanged run.

### Manifest

`run.json` is an object with these required fields:

| Field | Contract |
|---|---|
| `schema` | `nodus.goodput.run/0.2` |
| `suite` | `nodus/serve`, `nodus/train`, `nodus/kernels` |
| `run_id`, `arm` | Nonempty identifiers |
| `arm_class` | `nodus`, `baseline`, `reference` |
| `record` | `measured` or `imported`; MODEL and contextual records are rejected by the scorer |
| `seed`, `cohort` | Nonnegative integer; `development` or `holdout` |
| `validity` | `{ "passed": true }`; runner-specific checks remain in raw evidence |
| `correctness` | `{ "passed": true, "contract": "...", "evidence": REF }` |
| `provenance` | `{ "repository": "...", "commit": "<40 lowercase hex>", "sources": [REF, ...] }` |
| `configuration`, `events` | Each a REF |
| `environment` | Nonempty `gpu_model`, `driver`, `cuda`, `interconnect`, `image_digest` (`sha256:<64 hex>`); positive integer `gpu_count` |
| `allocations` | Nonempty array of `{ "id": "...", "gpu_count": 1, "started_s": ..., "released_s": ..., "release_confirmed": true }`; counts sum to cohort count |
| `window` | `{ "started_s": ..., "ended_s": ... }`, positive duration inside every allocation |
| `expected_events` | Positive number of offered requests, global training steps or paired kernel samples, not successful events only |
| `slo` | Serving only: positive `ttft_s` and `tpot_s` |
| `baseline` | Kernels only: production/vendor baseline identity with its pinned version |

REF is `{ "path": "relative/path", "sha256": "<64 lowercase hex>" }`. Every reference must resolve inside the bundle and match its SHA-256. Absolute paths, traversal, escaping symlinks and duplicate JSON keys are rejected. Source files are never modified.

The configuration is a hashed JSON object. `models` and `data` are nonempty arrays of `{id, revision, sha256}` pins; `software` is a nonempty name-to-version object. It also contains `precision`, `engine`, `warmup`, `search_budget`, `arm_order`, and `parallelism: {tensor: 1, pipeline: 1, data: 1}`. Record model/data content hashes, full engine parameters, batch/accumulation, memory caps, compiler options, search trials and rejected arms here. For synthetic kernel data, pin the generator plus shape manifest as data; a model pin may identify the production workload from which shapes came.

Parallelism metadata describes the executed configuration, including single GPU. Distributed serving should compare TP/PP and replicated engines at 1/2/4/8 GPUs when measured; training includes DDP/FSDP/ZeRO. Report communication/overlap/interconnect profiles as source artifacts. This scorer does not infer communication metrics from throughput or enforce an engine-specific TP×PP×DP topology equation.

### Events and metrics

`events` refers to JSONL with one object per event. IDs must be unique and count must match `expected_events`. Numeric fields reject booleans, nonfinite values and invalid chronology.

**Serving:** `{id, kind:"request", ok:bool, arrival_s, done_s, token_times_s:[...], output_tokens:int}`. Every offered request has a record, including errors/timeouts. Failed requests contribute zero goodput and remain in the attainment denominator. Successful requests need at least one token. TTFT is first-token time minus arrival; per-request TPOT is `(last-first)/(token_count-1)`, with one-token TPOT defined as zero. A request contributes output tokens only if it meets both targets. Report output tokens per allocated GPU-hour, SLO attainment and completion rate. `completed_e2e_p95_s` explicitly excludes failed requests and is always shown alongside those rates; it is not an all-request latency claim.

**Training:** `{id, kind:"step", validated:true, started_s, ended_s, tokens:int, loss:number}` in global-step order. Use global tokens across ranks once, not one duplicate per rank. Steps cannot overlap. Report validated tokens per allocated GPU-hour and steady tokens/s/GPU. There is no time-to-quality or convergence claim: a future quality-target analysis requires the held-out evaluation trace.

**Kernels:** `{id, kind:"kernel", correct:true, shape_id, candidate_ms, baseline_ms}`. One shape per run, positive paired timings, actual production baseline identity required. Report candidate median/p95, baseline median and ratio of medians. Compile/capture time is separate source evidence. A microbenchmark win makes no application-speedup claim.

These are per-run descriptive metrics, without significance intervals. Confirmatory comparisons require paired independent runs/seeds under the full protocol; do not treat request count as the number of independent experiments.

## Legacy evidence and scope

Legacy v0.1 data cannot be promoted by filling missing allocation timestamps or streamed token timings with guesses. It remains contextual under v0.2; existing v0.1 analysis remains authoritative for those runs.

The scorer has CPU contract checks using synthetic fixtures, not GPU measurements. Missing: GPU runners/adapters, independent hardware attestation, calibrated MODEL scoring, automated paired multi-run comparisons, fleet/elastic scoring, and the complete public execution bundle. A scored bundle is research evidence only; product integration is separate.