feat(runtime): Chronos-Bolt in forecasts, each series chosen once by the coordinator #668

2026-10-23T09:20:17Z

rvnminers-A-and-N commented

2026-10-23 09:20:17 +00:00

Part of #665. Builds on #667.

forecasts can now try the Chronos-Bolt kinds for each of many series, as forecast does for one: chronos-tiny, chronos-mini, chronos-small and chronos-base. They are opt-in, need a node that runs containers, and go only to workers that offer the same pinned image (the forecast-chronos:<hash> label from #667).

Who chooses each series' forecast

With Chronos-Bolt, a task no longer decides each series' forecast. For each series it returns:

the backtest windows;
the backtests the choice is made from.

Aggregate then decides, from the result taken. The backtests are the best candidate of each kind written in Go, plus every Chronos-Bolt candidate. decide uses nothing else from the Go kinds: it takes the best candidate, or the median of the best of each of the best kinds. A test checks that the kept candidates choose exactly as all of them would.

Two honest workers' Chronos-Bolt scores can differ in their last digits, and so can which candidate looks best. Choosing once, from one result, keeps that from becoming a disagreement between workers.

Lior, this differs from what we agreed on #665. There, Compare re-derived the choice from each worker's scores. Deciding in Aggregate gives the same guarantee and is simpler: there is no choice for Compare to check, only numbers within a tolerance. It is also how forecast already works. Are you happy with this?

Comparing

forecasts now Varies and Compares, series by series:
- everything except a series' candidates must match exactly;
- Go candidates must match to the bit;
- Chronos-Bolt candidates must agree within 1e-3, as in forecast.
compareCandidates is shared with forecast.
A job without Chronos-Bolt is still compared by its bytes, as before.

Since #669 (Lior's review items)

Per-series size. Each series' output carries its size, the largest |value| read from the input, alongside its candidates. compareCandidates(xs, ys, seriesSize) measures every Chronos-Bolt field against max(size, largest |forecast|). The size must match exactly, along with every other field of the series apart from its candidates.
Memory gate in forecasts too. forecasts' Split now uses the same cannotRunChronos as forecast. A test caught that it still checked only for containers. Offers, the form and the runners already shared it through f.forecast().
Regression tests. TestManySeriesAreComparedEachAgainstItsOwnSize covers:
- a residual 0.05 apart on a series of size 100 is accepted, and 0.2 apart is rejected;
- a larger size given by one output is caught;
- a size without candidates is refused;
- Execute writes each series' size;
- a node limited to 2048 MB neither offers nor splits Chronos-Bolt.

Other changes

outcomeOrder is factored out of decide, so that keptCandidates ranks a kind's candidates the same way.
Smaller tasks. A task with Chronos-Bolt holds fewer series, sized for the backtests each series carries.
Registry.Offers lists a label once. Before, forecast and forecasts both offered the image, so it was listed twice.
Form. The models field offers the Chronos-Bolt kinds only on a node that runs containers.

Tests

A many-series job with Chronos-Bolt forecasts each series as a forecast job of it alone would, checked step by step.
Compare: same bytes; within the tolerance; beyond it; a Go candidate differing; fields other than the candidates differing; series counts differing; outputs that are not JSON.
Malformed task outputs are refused by Aggregate. A series whose candidates all failed is counted as failed.
Needs and Offers labels, and the form choices.
Runtime coverage is complete apart from the docker engine default, which the realimages test covers.

Docs

docs/forecast-benchmark.md explains the coordinator's choice.
The security model and the README now say that forecast and forecasts with a Chronos-Bolt candidate are checked within a tolerance.

Part of #665. Builds on #667. `forecasts` can now try the Chronos-Bolt kinds for each of many series, as `forecast` does for one: `chronos-tiny`, `chronos-mini`, `chronos-small` and `chronos-base`. They are opt-in, need a node that runs containers, and go only to workers that offer the same pinned image (the `forecast-chronos:<hash>` label from #667). ### Who chooses each series' forecast With Chronos-Bolt, a task no longer decides each series' forecast. For each series it returns: - the backtest windows; - the backtests the choice is made from. `Aggregate` then decides, from the result taken. The backtests are the best candidate of each kind written in Go, plus every Chronos-Bolt candidate. `decide` uses nothing else from the Go kinds: it takes the best candidate, or the median of the best of each of the best kinds. A test checks that the kept candidates choose exactly as all of them would. Two honest workers' Chronos-Bolt scores can differ in their last digits, and so can which candidate looks best. Choosing once, from one result, keeps that from becoming a disagreement between workers. **Lior, this differs from what we agreed on #665.** There, `Compare` re-derived the choice from each worker's scores. Deciding in `Aggregate` gives the same guarantee and is simpler: there is no choice for `Compare` to check, only numbers within a tolerance. It is also how `forecast` already works. Are you happy with this? ### Comparing - `forecasts` now `Varies` and `Compare`s, series by series: - everything except a series' candidates must match exactly; - Go candidates must match to the bit; - Chronos-Bolt candidates must agree within 1e-3, as in `forecast`. - `compareCandidates` is shared with `forecast`. - A job without Chronos-Bolt is still compared by its bytes, as before. ### Since #669 (Lior's review items) - **Per-series size.** Each series' output carries its size, the largest |value| read from the input, alongside its candidates. `compareCandidates(xs, ys, seriesSize)` measures every Chronos-Bolt field against max(size, largest |forecast|). The size must match exactly, along with every other field of the series apart from its candidates. - **Memory gate in `forecasts` too.** `forecasts`' `Split` now uses the same `cannotRunChronos` as `forecast`. A test caught that it still checked only for containers. `Offers`, the form and the runners already shared it through `f.forecast()`. - **Regression tests.** `TestManySeriesAreComparedEachAgainstItsOwnSize` covers: - a residual 0.05 apart on a series of size 100 is accepted, and 0.2 apart is rejected; - a larger size given by one output is caught; - a size without candidates is refused; - `Execute` writes each series' size; - a node limited to 2048 MB neither offers nor splits Chronos-Bolt. ### Other changes - **`outcomeOrder`** is factored out of `decide`, so that `keptCandidates` ranks a kind's candidates the same way. - **Smaller tasks.** A task with Chronos-Bolt holds fewer series, sized for the backtests each series carries. - **`Registry.Offers`** lists a label once. Before, `forecast` and `forecasts` both offered the image, so it was listed twice. - **Form.** The models field offers the Chronos-Bolt kinds only on a node that runs containers. ### Tests - A many-series job with Chronos-Bolt forecasts each series as a `forecast` job of it alone would, checked step by step. - Compare: same bytes; within the tolerance; beyond it; a Go candidate differing; fields other than the candidates differing; series counts differing; outputs that are not JSON. - Malformed task outputs are refused by `Aggregate`. A series whose candidates all failed is counted as failed. - Needs and Offers labels, and the form choices. - Runtime coverage is complete apart from the `docker` engine default, which the realimages test covers. ### Docs - `docs/forecast-benchmark.md` explains the coordinator's choice. - The security model and the README now say that `forecast` and `forecasts` with a Chronos-Bolt candidate are checked within a tolerance.

excho0 commented

2026-10-23 09:25:25 +00:00

I’m happy with the coordinator-side choice in Aggregate. Given that candidate scores are compared within tolerance, choosing once from the accepted result avoids a second worker-specific choice at a near-tie boundary. I’ve updated my earlier note on #665 accordingly; the key invariant is that the chosen output comes from the same candidate-score payload that passed comparison, while the no-Chronos path stays byte-for-byte unchanged.

Two review items remain before I’d consider this complete: (1) please verify Chronos is not advertised/selected when the owner’s memory confinement cap is below its 4 GiB requirement (the current capability appears gated by container availability, not that cap); and (2) compareChronos should use a scale appropriate to each compared field—using forecast-output magnitude to set the tolerance for scores/residuals can make valid cross-hardware differences fail for near-zero forecasts or large residuals. Could you add focused regression cases for both?

/cc @rvnminers-A-and-N

I’m happy with the coordinator-side choice in `Aggregate`. Given that candidate scores are compared within tolerance, choosing once from the accepted result avoids a second worker-specific choice at a near-tie boundary. I’ve updated my earlier note on #665 accordingly; the key invariant is that the chosen output comes from the same candidate-score payload that passed comparison, while the no-Chronos path stays byte-for-byte unchanged. Two review items remain before I’d consider this complete: (1) please verify Chronos is not advertised/selected when the owner’s memory confinement cap is below its 4 GiB requirement (the current capability appears gated by container availability, not that cap); and (2) `compareChronos` should use a scale appropriate to each compared field—using forecast-output magnitude to set the tolerance for scores/residuals can make valid cross-hardware differences fail for near-zero forecasts or large residuals. Could you add focused regression cases for both? /cc @rvnminers-A-and-N

rvnminers-A-and-N commented

2026-10-23 09:28:25 +00:00

Holding this until #669 lands. #669 fixes the two points Lior raised on #667. Afterwards I'll rebase this PR and make forecasts give each series' size too, so its Chronos-Bolt candidates are compared against their own series.

Holding this until #669 lands. #669 fixes the two points Lior raised on #667. Afterwards I'll rebase this PR and make `forecasts` give each series' size too, so its Chronos-Bolt candidates are compared against their own series.

rvnminers-A-and-N commented

2026-10-23 09:30:04 +00:00

Thanks for confirming the Aggregate choice. Both review items are fixed in #669, which is on dev for forecast:

cannotRunChronos gates Offers, the form choices and the runners on the owner's memory ceiling as well as on containers, with tests for nodes limited to 4095 and 512 MB.
Each task gives its series' size, and compareChronos measures every field against max(series size, largest |forecast|). There are regression tests for a near-zero forecast with large residuals and for a size that differs between outputs.

Once #669 lands I'll rebase this PR. forecasts gets the same gate for free, since it uses the same Forecast runners. I'll also add a size for each series, so every series' Chronos-Bolt candidates are compared against their own series, with the same regression cases per series.

Thanks for confirming the Aggregate choice. Both review items are fixed in #669, which is on `dev` for `forecast`: 1. `cannotRunChronos` gates `Offers`, the form choices and the runners on the owner's memory ceiling as well as on containers, with tests for nodes limited to 4095 and 512 MB. 2. Each task gives its series' size, and `compareChronos` measures every field against max(series size, largest |forecast|). There are regression tests for a near-zero forecast with large residuals and for a size that differs between outputs. Once #669 lands I'll rebase this PR. `forecasts` gets the same gate for free, since it uses the same `Forecast` runners. I'll also add a size for each series, so every series' Chronos-Bolt candidates are compared against their own series, with the same regression cases per series.

rvnminers-A-and-N commented

2026-10-23 10:12:28 +00:00

Rebased onto 34aba6f (#669) as 3f0c03e. Both of your #668 review items are now covered for forecasts too:

Memory ceiling. forecasts' Split now uses cannotRunChronos. The new test caught it still checking only for containers. Offers, the form and the runners already shared the gate.
Per-field scale, per series. Each series carries its own input-derived size, and its Chronos-Bolt candidates are compared against max(that size, largest |forecast|). The size has to match exactly.

TestManySeriesAreComparedEachAgainstItsOwnSize is the regression case for both. I've updated the PR description to match.

Rebased onto `34aba6f` (#669) as `3f0c03e`. Both of your #668 review items are now covered for `forecasts` too: 1. **Memory ceiling.** `forecasts`' `Split` now uses `cannotRunChronos`. The new test caught it still checking only for containers. `Offers`, the form and the runners already shared the gate. 2. **Per-field scale, per series.** Each series carries its own input-derived size, and its Chronos-Bolt candidates are compared against max(that size, largest |forecast|). The size has to match exactly. `TestManySeriesAreComparedEachAgainstItsOwnSize` is the regression case for both. I've updated the PR description to match.

excho0 approved these changes

2026-10-23 10:15:26 +00:00

excho0 left a comment

Reviewed the rebased change against the two follow-ups from #667. forecasts now uses the same memory-confinement gate as forecast, carries each series’ input-derived scale with its candidate backtests, compares candidates per series, and leaves the coordinator to choose once from the accepted result. The registry now wires container capability and confinement into Forecasts; regression coverage exercises multi-series choice, per-series scales and the low-memory path. The docs describe the new behavior. I found no remaining review blocker. The CI is still running, so please merge only after every required check is green.

/cc @rvnminers-A-and-N

Reviewed the rebased change against the two follow-ups from #667. `forecasts` now uses the same memory-confinement gate as `forecast`, carries each series’ input-derived scale with its candidate backtests, compares candidates per series, and leaves the coordinator to choose once from the accepted result. The registry now wires container capability and confinement into `Forecasts`; regression coverage exercises multi-series choice, per-series scales and the low-memory path. The docs describe the new behavior. I found no remaining review blocker. The CI is still running, so please merge only after every required check is green. /cc @rvnminers-A-and-N

excho0 commented

2026-10-23 10:39:28 +00:00

The rebased #668 is now green: all 34 checks passed, including coverage, real-image tests, desktop E2E and packaging. My approval is in; it is ready for your squash merge.

/cc @rvnminers-A-and-N

The rebased #668 is now green: all 34 checks passed, including coverage, real-image tests, desktop E2E and packaging. My approval is in; it is ready for your squash merge. /cc @rvnminers-A-and-N

rvnminers-A-and-N commented

2026-10-23 10:43:14 +00:00

@excho0 Thanks for the review. Squash-merged as c32b53f; integration is synced and the R620 worker is on it. Next up is #670, the first dev/ batch of #633.

@excho0 Thanks for the review. Squash-merged as `c32b53f`; integration is synced and the R620 worker is on it. Next up is #670, the first `dev/` batch of #633.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Sisyphus/Sisyphus#668