Verify science workloads within a tolerance, not by identical bytes #627

New Issue

2026-10-21T17:15:11Z

rvnminers-A-and-N commented

2026-10-21 17:15:11 +00:00

Why

The pool checks work by running a task twice, with verify on other workers or audit_share on the coordinator, and comparing the bytes. That only works for workloads whose results are the same to the last bit on any machine, so every science workload so far refuses checking. Measurements:

md (OpenMM, #577): the same seed gives different energies between runs, even on one machine, because MD is chaotic.
qchem (xtb 6.7.1, #626): results are bit-identical on one machine, run after run and however the job is split. But different processors disagree: under emulated CPU models, the R620/Sandy Bridge, Haswell/Skylake (AVX2+FMA) and AMD EPYC each gave a different set of bits. The differences are only in the last digits: the HOMO-LUMO gap in the 11th decimal, the free energy in the 14th, coordinates in the 10th. Total energies to 12 decimals, and frequencies, agreed.
render, transcribe and ocr: these vary too, through denoising, model maths and version drift.

So exact comparison can't check useful science on a mixed pool. Comparing within a tolerance could, for any workload where honest results agree to a known precision. That's what proof of useful work (#589) needs.

What

An optional workload interface: Compare(a, b []byte) (same bool, why string). A workload that has it is checked with it instead of byte equality, and verify/audit_share stop refusing it.
qchem as the first user, with two tiers. Both tiers also require the same convergence and failure flags.
- Same CPU type (both results report the same CPU model): energies agree to 1e-8 Eh, gaps to 1e-6 eV, frequencies to 0.1 cm⁻¹ and coordinates to 1e-5 Å. xtb is bit-identical on one machine, so this tight tier catches small tampering.
- Different CPU types: the bounds that landed in #661, set from the measured cross-CPU drift: 1e-6 Eh, 1e-3 eV, 1 cm⁻¹ and 1e-3 Å coordinate RMSD.
- The comparator picks the tier and records which one settled the task (see 4). Either tier still catches fabricated numbers.
md later, by challenge on short windows, as planned on #577. That needs checkpoints first.
The coordinator records which comparison settled each task, so the contribution ledger (#580) can count it as verified.

Done when

A qchem job with verify: 2 succeeds when its two results come from different CPU types, and fails when one worker's energies are altered.

## Why The pool checks work by running a task twice, with `verify` on other workers or `audit_share` on the coordinator, and comparing the bytes. That only works for workloads whose results are the same to the last bit on any machine, so every science workload so far refuses checking. Measurements: - **`md` (OpenMM, #577):** the same seed gives different energies between runs, even on one machine, because MD is chaotic. - **`qchem` (xtb 6.7.1, #626):** results are bit-identical on one machine, run after run and however the job is split. But different processors disagree: under emulated CPU models, the R620/Sandy Bridge, Haswell/Skylake (AVX2+FMA) and AMD EPYC each gave a different set of bits. The differences are only in the last digits: the HOMO-LUMO gap in the 11th decimal, the free energy in the 14th, coordinates in the 10th. Total energies to 12 decimals, and frequencies, agreed. - **`render`, `transcribe` and `ocr`:** these vary too, through denoising, model maths and version drift. So exact comparison can't check useful science on a mixed pool. Comparing within a tolerance could, for any workload where honest results agree to a known precision. That's what proof of useful work (#589) needs. ## What 1. **An optional workload interface:** `Compare(a, b []byte) (same bool, why string)`. A workload that has it is checked with it instead of byte equality, and `verify`/`audit_share` stop refusing it. 2. **`qchem` as the first user, with two tiers.** Both tiers also require the same convergence and failure flags. - **Same CPU type** (both results report the same CPU model): energies agree to 1e-8 Eh, gaps to 1e-6 eV, frequencies to 0.1 cm⁻¹ and coordinates to 1e-5 Å. xtb is bit-identical on one machine, so this tight tier catches small tampering. - **Different CPU types:** the bounds that landed in #661, set from the measured cross-CPU drift: 1e-6 Eh, 1e-3 eV, 1 cm⁻¹ and 1e-3 Å coordinate RMSD. - The comparator picks the tier and records which one settled the task (see 4). Either tier still catches fabricated numbers. 3. **`md` later,** by challenge on short windows, as planned on #577. That needs checkpoints first. 4. **The coordinator records which comparison settled each task,** so the contribution ledger (#580) can count it as verified. ## Done when A `qchem` job with `verify: 2` succeeds when its two results come from different CPU types, and fails when one worker's energies are altered.

rvnminers-A-and-N commented

2026-10-22 23:18:06 +00:00

The mechanism landed in #629 (03c1556): runtime.Comparing, used by both verify and audit_share, with results grouped around the one that agrees with the most others. What's left here is each workload's own Compare, starting with qchem (xtb differs in the 10th to 14th decimal across processors), then md, then model-based forecasts.

The mechanism landed in #629 (03c1556): `runtime.Comparing`, used by both `verify` and `audit_share`, with results grouped around the one that agrees with the most others. What's left here is each workload's own `Compare`, starting with `qchem` (xtb differs in the 10th to 14th decimal across processors), then `md`, then model-based forecasts.

rvnminers-A-and-N commented

2026-10-23 00:36:45 +00:00

Before writing qchem's Compare I checked it against what a qchem task returns, and #629's mechanism can't reach it as things stand. There are two blockers:

The numbers are not in the output. A qchem task's output is a ContainerOutput: the CIDs of stdout, summary.json and structures.xyz. Compare(a, b []byte) sees only those CIDs and has no Blobs to open them with.
The CIDs differ anyway. Task.Same requires compared results to have stored the same blobs (verify.go:67). Across processors xtb's energies differ in the 11th decimal and its coordinates in the 10th, so summary.json and structures.xyz hash differently. Same returns "they stored different blobs" before Compare is ever called.

Options:

A. Put the numbers in the task's output, and store blobs only in Aggregate. qchem would run its container but return the summary and geometries inline, as JSON, instead of as files. Compare then parses both outputs and holds energies, gaps, dipoles, frequencies and coordinates to a tolerance. #629 stays as it is. The cost is output size: an XYZ structure is about 50 bytes per atom, so a run of 25 drug-sized conformers is around 100 KB. Inline outputs are already capped, so a job would be split into runs small enough to fit.
B. Give Compare the blobs (Compare(ctx, blobs, a, b)) and drop the same-blobs rule for comparing workloads. This is more general, and md's trajectories will need it too. But the coordinator would have to fetch both workers' blobs to compare them, and a result that verifies would still name blobs the other worker never stored.
C. Round before storing (e.g. coordinates to 6 decimals) so the CIDs agree. I don't recommend it: a value that sits on a rounding boundary still splits honest workers, rarely but unpredictably.

I lean to A for qchem now, since it needs no change to #629 and keeps verification on the coordinator cheap. B can come later, when md needs it. Does that match how you pictured it?

Before writing qchem's `Compare` I checked it against what a qchem task returns, and #629's mechanism can't reach it as things stand. There are two blockers: 1. **The numbers are not in the output.** A qchem task's output is a `ContainerOutput`: the CIDs of stdout, `summary.json` and `structures.xyz`. `Compare(a, b []byte)` sees only those CIDs and has no `Blobs` to open them with. 2. **The CIDs differ anyway.** `Task.Same` requires compared results to have stored the same blobs (`verify.go:67`). Across processors xtb's energies differ in the 11th decimal and its coordinates in the 10th, so `summary.json` and `structures.xyz` hash differently. `Same` returns "they stored different blobs" before `Compare` is ever called. Options: - **A. Put the numbers in the task's output, and store blobs only in `Aggregate`.** qchem would run its container but return the summary and geometries inline, as JSON, instead of as files. `Compare` then parses both outputs and holds energies, gaps, dipoles, frequencies and coordinates to a tolerance. #629 stays as it is. The cost is output size: an XYZ structure is about 50 bytes per atom, so a run of 25 drug-sized conformers is around 100 KB. Inline outputs are already capped, so a job would be split into runs small enough to fit. - **B. Give `Compare` the blobs** (`Compare(ctx, blobs, a, b)`) and drop the same-blobs rule for comparing workloads. This is more general, and md's trajectories will need it too. But the coordinator would have to fetch both workers' blobs to compare them, and a result that verifies would still name blobs the other worker never stored. - **C. Round before storing** (e.g. coordinates to 6 decimals) so the CIDs agree. I don't recommend it: a value that sits on a rounding boundary still splits honest workers, rarely but unpredictably. I lean to **A** for qchem now, since it needs no change to #629 and keeps verification on the coordinator cheap. B can come later, when md needs it. Does that match how you pictured it?

excho0 commented

2026-10-23 00:46:49 +00:00

Yes, A is how I pictured it for qchem. It keeps #629 untouched and verification on the coordinator cheap. B should wait until md actually needs it.

Two points for the Compare itself, so cross-processor noise doesn't become false mismatches:

Coordinates: compare by RMSD after alignment rather than atom by atom. Two honest optimisations can end in the same minimum with a tiny rigid-body drift, and a per-coordinate tolerance would reject them. Kabsch alignment over matching atom order is enough, since both workers start from the same input.
Frequencies: sort them and skip the near-zero rotational and translational modes before comparing. Those six modes are numerically noisy and their order can swap.

On sizing, deciding the split from the atom count up front (roughly 50 bytes per atom per conformer, against the inline cap) is simpler than retrying a run that overflows.

/cc @rvnminers-A-and-N

Yes, **A** is how I pictured it for qchem. It keeps #629 untouched and verification on the coordinator cheap. **B** should wait until md actually needs it. Two points for the `Compare` itself, so cross-processor noise doesn't become false mismatches: - **Coordinates:** compare by RMSD after alignment rather than atom by atom. Two honest optimisations can end in the same minimum with a tiny rigid-body drift, and a per-coordinate tolerance would reject them. Kabsch alignment over matching atom order is enough, since both workers start from the same input. - **Frequencies:** sort them and skip the near-zero rotational and translational modes before comparing. Those six modes are numerically noisy and their order can swap. On sizing, deciding the split from the atom count up front (roughly 50 bytes per atom per conformer, against the inline cap) is simpler than retrying a run that overflows. /cc @rvnminers-A-and-N

excho0 commented

2026-10-23 08:05:41 +00:00

qchem is now the first workload using the tolerance-based verification path: Austin’s #661 merged as 1fd8a53 after its full CI, including coverage and real-image tests, passed. Its task returns the numeric results inline for comparison; Compare accounts for aligned geometry and rotates dipoles into the same frame. This closes the qchem portion, not the tracking issue: MD and model-based forecasts still need workload-specific comparison, with the Chronos candidate work tracked in #665.

/cc @rvnminers-A-and-N

qchem is now the first workload using the tolerance-based verification path: Austin’s #661 merged as `1fd8a53` after its full CI, including coverage and real-image tests, passed. Its task returns the numeric results inline for comparison; `Compare` accounts for aligned geometry and rotates dipoles into the same frame. This closes the qchem portion, not the tracking issue: MD and model-based forecasts still need workload-specific comparison, with the Chronos candidate work tracked in #665. /cc @rvnminers-A-and-N

excho0 commented

2026-10-28 09:03:25 +00:00

One spec/implementation mismatch surfaced in the open-issue audit. The original “What” section proposes qchem tolerances of 1e-8 Eh, 1e-6 eV for the gap, 0.1 cm⁻¹ for frequencies, and 1e-5 Å for coordinates. The merged implementation in #661 instead uses 1e-6 Eh for energies/free energy, 1e-3 eV for gaps, 1 cm⁻¹ for frequencies, and 1e-3 Å aligned RMSD (plus dipoles/intensities). The PR explains these as measured-drift-based tolerances, so I’m not changing the spec unilaterally. Austin, can we confirm the landed bounds are the intended contract and then update the issue’s proposed values? Also, the Done-when’s cross-CPU qchem scenario should be distinguished from the generic comparator tests in #629. /cc @rvnminers-A-and-N

excho0 commented

2026-10-28 13:06:43 +00:00

Audit update: the Chronos forecast-comparison gap mentioned in the earlier status is now covered. Merged #667 adds tolerance-based Compare for the single-series forecast workload; #668 extends it to forecasts, comparing each series against its own scale. Both use the pinned Chronos image, and #665 records the accepted design and benchmark follow-up. So I am no longer treating Chronos model forecasts as outstanding in this issue.\n\nRemaining scope here is MD-specific verification (which needs its challenge/checkpoint design) and confirming the qchem tolerance contract: the proposed values in this issue differ from the measured bounds implemented in #661. I have not changed those values without Austin’s confirmation.\n\n/cc @rvnminers-A-and-N

Audit update: the Chronos forecast-comparison gap mentioned in the earlier status is now covered. Merged #667 adds tolerance-based `Compare` for the single-series `forecast` workload; #668 extends it to `forecasts`, comparing each series against its own scale. Both use the pinned Chronos image, and #665 records the accepted design and benchmark follow-up. So I am no longer treating Chronos model forecasts as outstanding in this issue.\n\nRemaining scope here is MD-specific verification (which needs its challenge/checkpoint design) and confirming the qchem tolerance contract: the proposed values in this issue differ from the measured bounds implemented in #661. I have not changed those values without Austin’s confirmation.\n\n/cc @rvnminers-A-and-N

rvnminers-A-and-N commented

2026-10-30 11:28:00 +00:00

Lior, thanks for holding this for me. My answer is both: two tiers, and I've updated the issue body to match.

Same CPU type: keep the original tight bounds (1e-8 Eh, 1e-6 eV, 0.1 cm⁻¹, 1e-5 Å). xtb is bit-identical on one machine, so anything looser there just gives a cheater room.
Different CPU types: the #661 bounds you measured (1e-6 Eh, 1e-3 eV, 1 cm⁻¹, 1e-3 Å RMSD). Those are the honest cross-CPU spread.

The comparator would pick the tier from the CPU model each worker reports, and record which tier settled the task for the ledger (#580). Agreed on keeping the cross-CPU qchem "Done when" separate from the generic comparator tests in #629.

One question: do results already carry the worker's CPU model somewhere the comparator can see, or does that need a small addition? If it needs one, I'm happy to write it.

Lior, thanks for holding this for me. My answer is both: two tiers, and I've updated the issue body to match. - **Same CPU type:** keep the original tight bounds (1e-8 Eh, 1e-6 eV, 0.1 cm⁻¹, 1e-5 Å). xtb is bit-identical on one machine, so anything looser there just gives a cheater room. - **Different CPU types:** the #661 bounds you measured (1e-6 Eh, 1e-3 eV, 1 cm⁻¹, 1e-3 Å RMSD). Those are the honest cross-CPU spread. The comparator would pick the tier from the CPU model each worker reports, and record which tier settled the task for the ledger (#580). Agreed on keeping the cross-CPU qchem "Done when" separate from the generic comparator tests in #629. One question: do results already carry the worker's CPU model somewhere the comparator can see, or does that need a small addition? If it needs one, I'm happy to write it.

excho0 commented

2026-10-30 17:05:02 +00:00

I checked the current dev implementation: no, the comparator cannot see the worker's CPU model today. worker.Hardware includes CPUModel, but pb.TaskResult carries only the task/attempt, outcome, blob references and model-use data; runtime.Comparing.Compare receives only the two output byte slices. So choosing the same-CPU vs cross-CPU qchem tolerance tier needs an addition.

I would avoid trusting a CPU model embedded in the output itself, since the worker controls that payload. The coordinator should associate each result with the hardware metadata of the worker that executed that attempt and provide that context to the comparator. Since you offered to write this piece, would you like to take the protocol/coordinator change? I’ll hold off on overlapping work until we coordinate.

/cc @rvnminers-A-and-N

I checked the current `dev` implementation: no, the comparator cannot see the worker's CPU model today. `worker.Hardware` includes `CPUModel`, but `pb.TaskResult` carries only the task/attempt, outcome, blob references and model-use data; `runtime.Comparing.Compare` receives only the two output byte slices. So choosing the same-CPU vs cross-CPU qchem tolerance tier needs an addition. I would avoid trusting a CPU model embedded in the output itself, since the worker controls that payload. The coordinator should associate each result with the hardware metadata of the worker that executed that attempt and provide that context to the comparator. Since you offered to write this piece, would you like to take the protocol/coordinator change? I’ll hold off on overlapping work until we coordinate. /cc @rvnminers-A-and-N

rvnminers-A-and-N commented

2026-10-30 17:16:02 +00:00

Thanks for checking. Agreed: the CPU model has to come from the coordinator, never from the worker's own output. I'll take the protocol/coordinator change.

Rough shape:

The coordinator records, with each result, the hardware of the worker that ran that attempt. It uses the CPUModel it already has from worker.Hardware at assignment time.
The comparator gets that context. Either Compare grows a small context argument, or there's a sibling interface, so workloads that don't care are untouched.
qchem picks the tight tier only when both CPU models are known and equal, and falls back to the cross-CPU tier when either is missing.
The tier that settled the task goes into the task record for #580.

I'll put it up as its own PR once #771 (upscaling) is in, and tag you on it.

Thanks for checking. Agreed: the CPU model has to come from the coordinator, never from the worker's own output. I'll take the protocol/coordinator change. Rough shape: - The coordinator records, with each result, the hardware of the worker that ran that attempt. It uses the `CPUModel` it already has from `worker.Hardware` at assignment time. - The comparator gets that context. Either `Compare` grows a small context argument, or there's a sibling interface, so workloads that don't care are untouched. - qchem picks the tight tier only when both CPU models are known and equal, and falls back to the cross-CPU tier when either is missing. - The tier that settled the task goes into the task record for #580. I'll put it up as its own PR once #771 (upscaling) is in, and tag you on it.

rvnminers-A-and-N commented

2026-10-30 20:51:56 +00:00

@excho0 #773 implements the two tiers. The coordinator takes each worker's CPU model from the Hello it joined with (worker.Hardware.CPUModel), never from the output. It notes it when it hands out the attempt, and the comparator only uses the tight tier when both models are known and equal. The tier is stored per task (tasks.tier) for #580. Four open questions are in the PR body: whether to add the tier to the signed job record, the tight dipole/intensity values I picked, recording mixed tiers as cross-cpu+same-cpu, and the real two-CPU xtb run that "Done when" still needs.

@excho0 #773 implements the two tiers. The coordinator takes each worker's CPU model from the Hello it joined with (`worker.Hardware.CPUModel`), never from the output. It notes it when it hands out the attempt, and the comparator only uses the tight tier when both models are known and equal. The tier is stored per task (`tasks.tier`) for #580. Four open questions are in the PR body: whether to add the tier to the signed job record, the tight dipole/intensity values I picked, recording mixed tiers as `cross-cpu+same-cpu`, and the real two-CPU xtb run that "Done when" still needs.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Sisyphus/Sisyphus#627