Molecular dynamics workload (OpenMM), replicas across the pool #577

New Issue

2026-10-21T00:30:27Z

rvnminers-A-and-N commented

2026-10-21 00:30:27 +00:00

Why

Molecular dynamics is real scientific work, and it splits naturally. Many independent simulations can run side by side and be combined afterwards: replicas, different starting conditions, or windows along a reaction path. That makes it a good first workload beyond the demos.

What

An md workload that runs OpenMM in a container the worker builds itself, the way transcode does.
- OpenMM is MIT or LGPL licensed.
- Use CUDA where the worker has a card, and CPU otherwise.
Input:
- a system (a PDB or an OpenMM XML, by CID);
- a force field;
- steps;
- temperature;
- how many replicas.
Each task is one replica with its own seed.
Output, per task:
- the final state;
- the energies;
- a trajectory, by CID.
The job's result summarises the replicas: mean energy, spread, and the trajectories' CIDs.
Verification: OpenMM with a fixed seed and the reference platform is reproducible, so the existing cross-checks can run on a sample of tasks.

First run

Alanine dipeptide in water, 32 replicas, across the R620 (CPU) and the laptop (CUDA).

Done when

The planner can be asked for it in words ("simulate alanine dipeptide for 1 ns, 32 replicas") and the summary comes back correct on both machines.

## Why Molecular dynamics is real scientific work, and it splits naturally. Many independent simulations can run side by side and be combined afterwards: replicas, different starting conditions, or windows along a reaction path. That makes it a good first workload beyond the demos. ## What - An `md` workload that runs OpenMM in a container the worker builds itself, the way `transcode` does. - OpenMM is MIT or LGPL licensed. - Use CUDA where the worker has a card, and CPU otherwise. - Input: - a system (a PDB or an OpenMM XML, by CID); - a force field; - steps; - temperature; - how many replicas. Each task is one replica with its own seed. - Output, per task: - the final state; - the energies; - a trajectory, by CID. The job's result summarises the replicas: mean energy, spread, and the trajectories' CIDs. - Verification: OpenMM with a fixed seed and the reference platform is reproducible, so the existing cross-checks can run on a sample of tasks. ## First run Alanine dipeptide in water, 32 replicas, across the R620 (CPU) and the laptop (CUDA). ## Done when The planner can be asked for it in words ("simulate alanine dipeptide for 1 ns, 32 replicas") and the summary comes back correct on both machines.

rvnminers-A-and-N commented

2026-10-21 01:12:41 +00:00

Plan for after #587, following the owner's direction: md should survive workers dropping out, and its work should still be checked.

What I measured on the R620. I ran the villin system in OpenMM 8.2 on the CPU platform, 250 steps, with the same seed every time:

Run	Final potential energy, kJ/mol
2 threads, run 1	−114,854.59
2 threads, run 2	−114,933.94
4 threads	−115,748.57

The results differ even with DeterministicForces on and without minimisation. MD is chaotic, so tiny rounding differences grow within a picosecond. Checking by exact comparison can't work for md, and #587 refuses verify and audit_share for that reason. A check has to tolerate small differences, and it has to look over a short stretch only.

Fault tolerance: long replicas as chained segments

A long replica is split into segments, say 100 ps each. Each segment is a task that starts from the previous segment's checkpoint (OpenMM's saveState, stored by CID) and leaves its own.
If a worker drops out, only its current segment reruns, not the whole replica. The chain may move between workers as segments finish.

Checking: commit, then challenge

Each segment reports the hash of every frame it saved, along with its checkpoints, so the worker has committed to the whole trajectory before anyone picks what to check.
A verifier, which is another worker or the coordinator, then picks a frame at random. It reruns a short window from that frame's checkpoint (a few dozen steps, with the same seed) and compares the result with the next saved frame. Over so few steps honest runs agree to a small tolerance, while a faked trajectory can't match.
It also checks the physics: the temperature should be near the target, the energy within range, and the molecule still in one piece.
This needs a new kind of verification, where a workload supplies its own comparison with a tolerance. That is a coordinator change, and the place where #580's ledger learns that md work is verified.

More of what people use

Amber inputs: accept a prmtop and inpcrd pair. OpenMM reads them directly and uses the same force fields.
An AmberTools image (free, open source, from conda-forge) for tleap preparation and cpptraj analysis as workloads of their own.
CUDA: OpenMM's CUDA plugin in the image, and jobs that ask for a GPU (the laptop's 4050).

pmemd, Amber's own fast engine, and Gaussian are licensed software. We can't ship them in an image. Gaussian's licence also ties it to the licensee's own machines, so running it on other people's workers would break it. For quantum chemistry, open programs fill that role: Psi4 (LGPL), xtb (LGPL), PySCF (Apache-2.0) and NWChem (ECL-2.0). Those are now on #584.

Plan for after #587, following the owner's direction: md should survive workers dropping out, and its work should still be checked. **What I measured on the R620.** I ran the villin system in OpenMM 8.2 on the CPU platform, 250 steps, with the same seed every time: | Run | Final potential energy, kJ/mol | |---|---| | 2 threads, run 1 | −114,854.59 | | 2 threads, run 2 | −114,933.94 | | 4 threads | −115,748.57 | The results differ even with `DeterministicForces` on and without minimisation. MD is chaotic, so tiny rounding differences grow within a picosecond. **Checking by exact comparison can't work for md**, and #587 refuses `verify` and `audit_share` for that reason. A check has to tolerate small differences, and it has to look over a short stretch only. **Fault tolerance: long replicas as chained segments** - [ ] A long replica is split into segments, say 100 ps each. Each segment is a task that starts from the previous segment's checkpoint (OpenMM's `saveState`, stored by CID) and leaves its own. - [ ] If a worker drops out, only its current segment reruns, not the whole replica. The chain may move between workers as segments finish. **Checking: commit, then challenge** - [ ] Each segment reports the hash of every frame it saved, along with its checkpoints, so the worker has committed to the whole trajectory before anyone picks what to check. - [ ] A verifier, which is another worker or the coordinator, then picks a frame at random. It reruns a short window from that frame's checkpoint (a few dozen steps, with the same seed) and compares the result with the next saved frame. Over so few steps honest runs agree to a small tolerance, while a faked trajectory can't match. - [ ] It also checks the physics: the temperature should be near the target, the energy within range, and the molecule still in one piece. - [ ] This needs a new kind of verification, where a workload supplies its own comparison with a tolerance. That is a coordinator change, and the place where #580's ledger learns that md work is verified. **More of what people use** - [ ] Amber inputs: accept a `prmtop` and `inpcrd` pair. OpenMM reads them directly and uses the same force fields. - [ ] An AmberTools image (free, open source, from conda-forge) for `tleap` preparation and `cpptraj` analysis as workloads of their own. - [ ] CUDA: OpenMM's CUDA plugin in the image, and jobs that ask for a GPU (the laptop's 4050). `pmemd`, Amber's own fast engine, and Gaussian are licensed software. We can't ship them in an image. Gaussian's licence also ties it to the licensee's own machines, so running it on other people's workers would break it. For quantum chemistry, open programs fill that role: Psi4 (LGPL), xtb (LGPL), PySCF (Apache-2.0) and NWChem (ECL-2.0). Those are now on #584.

rvnminers-A-and-N commented

2026-10-21 04:34:28 +00:00

First real run, 2026-10-21. The owner started it from the desktop Run page on the Windows laptop, which coordinates the trial pool. It ran on the R620 worker, using containers on the CPU. The molecule was villin in water (8,867 atoms), simulated as 4 replicas of 5 ps each:

Replica	Seed	ns/day	Final potential energy (kJ/mol)
0	1	4.50	−115,702.8
1	2	4.16	−116,228.3
2	3	4.57	−116,787.3
3	4	4.61	−116,677.0

Mean: −116,348.9 kJ/mol, with a spread of 427.7. Each replica took 3 min 16 s to 3 min 50 s on the R620, all four at once while CI jobs shared the machine.
Files: each replica's last frame, energies and trajectory came back by CID and can be downloaded. #598 makes them save as .pdb, .csv and .dcd.
Next, in the order planned above:
1. a result view on the Run page, with a table and an energy plot, instead of the raw JSON;
2. CUDA, once a GPU worker runs containers (#596 on the laptop);
3. then chained segments with checkpoints, and checking by challenge.

**First real run, 2026-10-21.** The owner started it from the desktop Run page on the Windows laptop, which coordinates the trial pool. It ran on the R620 worker, using containers on the CPU. The molecule was villin in water (8,867 atoms), simulated as 4 replicas of 5 ps each: | Replica | Seed | ns/day | Final potential energy (kJ/mol) | |---|---|---|---| | 0 | 1 | 4.50 | −115,702.8 | | 1 | 2 | 4.16 | −116,228.3 | | 2 | 3 | 4.57 | −116,787.3 | | 3 | 4 | 4.61 | −116,677.0 | - **Mean:** −116,348.9 kJ/mol, with a spread of 427.7. Each replica took 3 min 16 s to 3 min 50 s on the R620, all four at once while CI jobs shared the machine. - **Files:** each replica's last frame, energies and trajectory came back by CID and can be downloaded. #598 makes them save as `.pdb`, `.csv` and `.dcd`. - **Next, in the order planned above:** 1. a result view on the Run page, with a table and an energy plot, instead of the raw JSON; 2. CUDA, once a GPU worker runs containers (#596 on the laptop); 3. then chained segments with checkpoints, and checking by challenge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Sisyphus/Sisyphus#577