# Nodus GPU Goodput: public protocol, draft 0.2

Status: protocol and evidence scorer draft. No new v0.2 GPU results are claimed.
IdleBench v0.1 remains a separate version with its existing measurements and denominator.

## Primary outcome

Useful work per allocated GPU-hour under fixed correctness, quality and service contracts.
Count every allocation from start through confirmed release, including idle time and transitions.
Report steady-state window throughput separately. GPU busy percentage is a diagnostic, not the objective.
Do not aggregate incompatible tasks into one score.

## Suites and distributed scope

- nodus/serve: output tokens from requests meeting both time-to-first-token (TTFT) and mean time-per-output-token (TPOT) targets, per allocated GPU-hour. Count failed and timed-out requests in offered-load attainment. Compare pinned stock and tuned stock engines with equal search budgets. Record batching, prefix caching, supported precision and speculation separately and verify quality.
- nodus/train: validated global tokens per allocated GPU-hour. Hold model, data order, optimizer and global batch fixed. Time to a predeclared held-out quality target requires actual evaluation events; step throughput alone does not establish convergence.
- nodus/kernels: paired median/tail device timing and correctness against the actually dispatched production/vendor baseline. Freeze shapes, layouts, strides, precision and tolerances. Record compile/graph-capture costs separately. Integrate winners into a measured application before claiming application speedup.
- idle/s1–s4: existing IdleBench wake, model multiplexing, complementary work and recovery contracts remain v0.1.
- fleet: intended allocation-efficiency studies. Calibrated MODEL projections remain separate from measured fleet observations. Fleet/elastic scoring is not implemented in v0.2.

Evaluate single GPUs and 1/2/4/8-GPU cohorts where hardware is available. Serving includes tensor/pipeline parallelism and replicas; training includes DDP/FSDP/ZeRO. Report interconnect, topology, communication time and overlap from recorded profiles. Strong scaling fixes global work/batch; weak scaling gets a separate table. Missing and unsupported cells are explicit, never simulated as measurements.

## Freeze the experiment

Before execution record source commit, image digest, model/data revisions and hashes, driver/CUDA/library/compiler versions, GPU model/count/interconnect, engine parameters, precision, memory limits, traces, seeds, quality tolerances, SLOs, warm-up, window, arm order and tuning budget. Keep all rejected configurations. Compare on the same workload and resource constraints.

Freeze candidate selection on development seeds and evaluate held-out seeds once. A later candidate becomes a new version. Confirmatory claims require at least five independent paired runs/seeds with uncertainty over independent runs; smaller samples are labelled pilots. Request-level resampling cannot substitute for independent runs. Publish losses, invalid runs, OOMs, failures and missing cases beside wins. Never change SLOs after seeing results.

## Evidence and imports

Each scored run needs hash-bound configuration, raw events, correctness evidence and provenance. The draft contract describes `nodus.goodput.run/0.2`; the accompanying Python standard-library scorer checks structure, hashes, event accounting and allocation completion. It does not independently attest hardware or correctness.

Keep measured, imported and calibrated MODEL evidence distinct. Imports retain original units, parameters, provenance and limitations. Legacy records lacking actual streamed token timestamps or complete allocation lifetimes remain contextual under v0.2; never invent fields to promote them. Original v0.1 analysis remains authoritative for v0.1.

Mixed serving/training should report a vector of service goodput/attainment and validated training progress against each isolated reference. Any normalized sum requires both service and correctness gates and an explicit definition. Kernels are namespaced as nodus/kernels, distinct from the separately maintained ScalingIntelligence/KernelBench project.

## Public release and product claims

A complete reproducibility release needs the pinned execution kit, inputs, baseline configurations, all raw evidence and analysis. This release currently provides the draft contract/scorer and an unchanged IdleBench s2 result table; the full GPU execution bundle and raw-event archive are pending.

A scored record is research evidence, not a novelty verdict or a product-release claim. Track measured research, runtime candidates, validated changes, released features and organization enablement separately.