Sisyphus/apps/sisyphusd/README.md

# sisyphusd

The Sisyphus node daemon. One binary runs the coordinator role, the worker role, or both, and doubles as the command-line client.

A coordinator splits a job into tasks, workers execute them, and the results are aggregated. Around that are the pieces a pool needs: who may connect, storage that outlasts a job, private and verified jobs, containers, a planner, and a server for AI agents. What is built and what is not is on the [roadmap](https://github.com/sisyphus-network/Sisyphus/issues/23); what it protects and what it does not is in the [security model](../../docs/security-model.md).

## Build

Needs Go 1.27 or newer.

From the repository root, `nix develop` provides the Go toolchain and development dependencies. The daemon is the Go binary at `bin/sisyphusd`.

```sh
make build   # produces bin/sisyphusd
make test    # go vet and the test suite with the race detector
make cover   # fails unless the tests execute every statement of hand-written code
```

## Releases

Pushing a tag such as `v0.1.0` makes GitHub build `sisyphusd` and publish the binaries, with checksums, as a release. `scripts/build-release.sh <version> <dir>` does the same build locally.

| System | Processors |
| --- | --- |
| Linux | x86-64, ARM64, 32-bit ARM (v7), RISC-V 64 |
| macOS | Intel, Apple silicon |
| Windows | x86-64, ARM64 |
| FreeBSD | x86-64, ARM64 |
| OpenBSD, NetBSD | x86-64 |
| Android | ARM64, for use in a terminal such as Termux |

Only Linux on x86-64 is run by the test suite. The others are compiled and have not been run. Solaris, illumos, AIX and Plan 9 are not built because a dependency does not support them.

## Try it on one machine

The quick way, in one terminal, is `make demo`: it starts three nodes, runs a few jobs across them, and shuts them down.

To run the nodes yourself, use one terminal per node (or one `tmux` pane each). Every node keeps its key and data under `--data-dir`, so nodes sharing a machine each need their own. Left to itself a node uses the place its operating system sets aside for a program's data (`~/.local/share/sisyphus` on Linux, `~/Library/Application Support/sisyphus` on macOS, `%LOCALAPPDATA%\sisyphus` on Windows), or `~/.sisyphus` if that is already there from an earlier version. `sisyphusd data-dir` prints which.

```sh
# Terminal 1: a coordinator that is also a worker, using the default data directory
bin/sisyphusd run --name alpha --slots 4

# Terminal 2: ask alpha for an invitation, then start a second worker with it
bin/sisyphusd pool invite
bin/sisyphusd run --role worker --coordinator 127.0.0.1:7700 --join <invitation> \
    --name beta --slots 4 --data-dir ~/.sisyphus-beta

# Terminal 3
bin/sisyphusd nodes
bin/sisyphusd job submit --params '{"from":0,"to":3000000000}'
bin/sisyphusd job submit --mode full-worker --params '{"from":0,"to":3000000000}'
```

The first job is split across both workers; the second runs whole on one. Stop a worker with Ctrl-C while a job is running and its tasks move to the other. A worker needs `--join` only the first time. After that it remembers its coordinator, and an invitation left on its command line is ignored, so the same command restarts it.

## Try it across machines

On the coordinator machine, listen on an address the others can reach, and issue an invitation for each worker:

```sh
bin/sisyphusd run --listen 0.0.0.0:7700
bin/sisyphusd pool invite        # once per worker; each invitation works once
```

On each other machine:

```sh
bin/sisyphusd run --role worker --coordinator <coordinator-ip>:7700 --join <invitation>
```

To submit jobs from another machine, invite it as a client instead and join from there:

```sh
bin/sisyphusd pool invite --role client                       # on the coordinator
bin/sisyphusd pool join --addr <coordinator-ip>:7700 <invitation>   # on the other machine
bin/sisyphusd nodes --addr <coordinator-ip>:7700
```

## Who can connect

Every connection between nodes is encrypted, and each end proves who it is.

**Identity.** A node has a key pair, created the first time it runs and kept in `node.key` in its data directory, readable only by its owner. Its ID is derived from the public key and starts `12D3KooW`; `sisyphusd id` prints it. The name given with `--name` is only a label. Losing `node.key` loses the identity, and copying it to another machine gives that machine the same one.

**Encryption.** Connections use TLS 1.3. Each node presents a certificate signed with its own key, with no certificate authority involved: a node is recognised by its ID, not by a host name.

**Roles.** A node answers a caller according to what the caller has been admitted as:

| Role | Who | May |
| --- | --- | --- |
| owner | Whoever holds the node's own key: its own worker, and commands run from its data directory | Everything, including inviting and removing others |
| worker | A node invited with `pool invite` | Take tasks, fetch any stored blob it can name and store blobs, be given the key of the pool's private network, and publish its own name |
| client | A node invited with `pool invite --role client` | Submit jobs, and watch and cancel the ones it submitted; list nodes; store data, and pin and release its own pins. One client does not see another's jobs or pins |

Anyone else can complete the handshake and is then refused.

**How many connections.** A node takes as many connections at once as half the files its system lets a program have open, which on a server is hundreds of thousands and on a laptop is still thousands; `--max-connections` sets another number. An eighth of them, and never fewer than 256, may come from one address, since the machines of a house or an office share one, and each command run from another machine is a connection of its own. New connections from one address are slowed only after that many have come in a row. Raise the system's own limit (`ulimit -n`) to raise the node's.

**Joining.** An invitation is the inviting node's ID and a one-time token. The ID is what makes joining safe: the joiner will only talk to the node named in it, so nobody in between can pose as that node. It lasts an hour unless `--ttl` says otherwise, and is still good if the inviting node restarts before it is used. Once joined, a node remembers which node it expects at that address in `known.json` and refuses any other.

```sh
bin/sisyphusd pool members            # who has been admitted, and as what
bin/sisyphusd pool remove <node-id>   # take a node off the list and disconnect it
bin/sisyphusd pool rekey              # change the key of the pool's private IPFS network
bin/sisyphusd pool cluster            # with --cluster: which nodes hold each thing pinned
```

## Storing data

Every node keeps a content-addressed store under `--data-dir`. A coordinator's store is durable: a blob it has accepted survives a crash or power loss. A worker-only node's store is a cache of what its coordinator holds. It skips waiting for the disk on each write, which makes fetching several times faster on a slow disk, and it is kept across restarts. Because a power cut can then leave a cached block incomplete, a worker checks a cached blob in full the first time it is used in each run and downloads it again if it does not hold up. On a fast disk, `--sync-cache` makes the cache wait for the disk after all, so that it survives a power cut without downloading again. With `--max-cache-bytes`, a worker evicts cached blobs once the cache passes that size: first those it has not used since it started, then the least recently used, and never one a task has open. The `blob` commands talk to a node running the coordinator role.

```sh
bin/sisyphusd blob put results.tar     # prints the blob's CID
bin/sisyphusd blob stat <cid>
bin/sisyphusd blob get -o copy.tar <cid>
```

A blob's CID is the one `ipfs add --cid-version=1` gives the same file, and blocks are laid out on disk the way Kubo lays them out, so data can move to IPFS later without being renamed. Both `put` and `get` check the bytes against the CID, so a node cannot substitute different data unnoticed.

Jobs pass large data by CID instead of inline. A worker downloads an input from its coordinator the first time a task needs it, checks it against the CID, and keeps it; whatever a task stores is uploaded to the coordinator before the task reports success.

```sh
cid=$(bin/sisyphusd blob put big.txt)
bin/sisyphusd job submit --workload wordcount --params "{\"input\":\"$cid\"}"
bin/sisyphusd blob get <output CID from the result> | head
```

### Workers serving each other

A worker normally downloads a job's input from its coordinator. Started with `--serve`, it also lets the other workers of its pool download from it what it has cached, and every worker asks the coordinator who else holds a blob before asking the coordinator for the blob itself:

```sh
bin/sisyphusd run --role worker --coordinator <addr> --join <invitation> \
    --serve 0.0.0.0:7701 --advertise <this-machine's-address>:7701
```

- `--serve` is where it listens; `--advertise` is the address other workers are told, needed when `--serve` names every interface.
- A worker serves only nodes the coordinator confirms are members of the pool, only reads, and over the same encrypted, identified connections as everything else. What it receives is checked against its CID, so a worker cannot pass off different data.
- The coordinator's knowledge of who holds what comes from which workers ran tasks that used a blob. It is a hint; if no worker named can supply the blob, the coordinator does.
- A node removed from the pool may be served by a worker for up to a minute more, which is how long a worker remembers that a node is a member.
- This needs no Kubo. With `--kubo` the pool's private IPFS network does the same job and is tried first.

### Kubo and the pool's private IPFS network

By default a node keeps blocks in files of its own and sends them to other nodes itself. With `--kubo` it instead starts [Kubo](https://github.com/ipfs/kubo), the main IPFS implementation, beside itself, keeps its blocks there, and lets the pool's Kubo daemons exchange them directly. The `ipfs` program must be installed on every node that uses it.

```sh
bin/sisyphusd run --kubo                                   # the coordinator
bin/sisyphusd run --kubo --role worker --coordinator <addr> --join <invitation>
IPFS_PATH=$(bin/sisyphusd data-dir)/ipfs ipfs cat <cid>                  # anything stored is an ordinary IPFS file
```

- **Each node's Kubo is that node.** `ipfs id` and `sisyphusd id` print the same ID. Its repository is `ipfs` in the node's data directory, and it is started and stopped with the node.
- **The pool has a network of its own.** A coordinator started with `--kubo` creates a swarm key, kept in `swarm.key` in its data directory. Only Kubo daemons holding that key can connect to each other; they have no bootstrap peers, no gateway, and no way to reach or be reached from the public IPFS network.
- **Workers are given the key when they start**, over their encrypted connection to the coordinator, along with where to find the coordinator's Kubo and the other members'. A worker using `--kubo` needs a coordinator that does.
- **Data comes from whoever has it.** When a task opens a blob its worker does not hold, the worker's Kubo fetches the blocks from the members that do, the coordinator or other workers. If the network cannot supply it within thirty seconds, the worker falls back to asking the coordinator directly, and logs a warning saying so: the job still succeeds, but that warning means the private network is not working for that worker.
- **Removing a member changes the key.** A removed node still holds the key it was given, so `pool remove` gives the network a new one. The coordinator's Kubo restarts on it, which takes a few seconds, and tells the remaining workers, who fetch the new key and restart theirs. The removed node cannot ask for it. `pool rekey` does the same without removing anyone, for when a key may have leaked.
- **It needs no port of its own.** The coordinator's Kubo listens to its own machine only. A worker's Kubo reaches it through the connection the worker already has with the coordinator on `--listen`, so a pool needs one open port in all, and a worker needs none. See [One port](#one-port) below.
- **Kubo's API is open to this machine unless you close it.** It listens on a loopback port, and any program on the machine that finds the port can read what is stored unsealed, and pin and unpin. `--kubo-api-secret` has it answer only a caller that shows a secret, which the node makes and keeps in `kubo.token` in its data directory. The `ipfs` command must then be given it: `IPFS_PATH=$(bin/sisyphusd data-dir)/ipfs ipfs --api-auth "basic:sisyphus:$(cat "$(bin/sisyphusd data-dir)/kubo.token")" cat <cid>`. It cannot be used with `--cluster`, whose peer pins through the same API and cannot show a secret. Each node chooses for itself: a pool may mix nodes that use it with nodes that do not.
- Files added with the `ipfs` command on a node's repository can be used as job inputs by CID.
- The node's own pins decide what it keeps, as described below. Anything pinned with `ipfs pin add` is also kept. Do not run `ipfs repo gc` on the repository: Kubo's collector does not know about the node's pins.
- Moving an existing node to `--kubo`, or back, does not carry its stored data across.

### IPFS Cluster: pins held by several nodes

With `--kubo` alone, the coordinator's disk holds the only copy of the pool's data that anything promises to keep; what workers hold is a cache. With `--cluster` as well, each node also runs an [IPFS Cluster](https://ipfscluster.io) peer beside its Kubo, the pool's peers form one cluster, and whatever the coordinator pins is pinned on several nodes. The `ipfs-cluster-service` program must be installed on every node that uses it.

```sh
bin/sisyphusd run --kubo --cluster --replicas 1             # the coordinator: one copy besides its own
bin/sisyphusd run --kubo --cluster --role worker --coordinator <addr> --join <invitation>
bin/sisyphusd pool cluster                                  # which nodes hold what
```

- **The node's pins are the cluster's pins.** Everything in `blob pins`, apart from the hour's grace every upload gets and the records of jobs, which are not files, is pinned on the cluster, and unpinned from it when the last pin on the node goes: unpinned by hand, or lapsed and collected. Retention is still managed in the one place, with `blob pin`, `blob unpin`, `--retain` and `--ttl`. The coordinator compares the two lists when it starts and every minute after, and puts right whatever differs.
- **`--replicas N` is how many nodes hold each pin besides the coordinator**, one unless told otherwise; `pool cluster` counts the coordinator too, so it shows N + 1. The coordinator always holds everything: its Kubo is where the node stores it in any case. The others are chosen by the cluster, by free space, and fetch the data over the pool's private network.
- **With fewer nodes than that, a pin is held by as many as there are**, and gains holders within about a minute of more joining.
- **When a holder goes, another takes its place.** A node not heard from for thirty seconds is taken to be gone, and the coordinator's peer has another node fetch what it held, if there is one that does not hold it already. If the node comes back, the pin stays where it was moved to.
- **Only the coordinator says what is pinned.** A worker's peer is a follower: it keeps what it is allocated, refuses to pin or unpin anything itself, and every peer ignores changes that do not come from the coordinator's. Someone who gives a pool disk space gives it nothing else.
- **The cluster is closed as the private network is.** Its secret is derived from the swarm key, so nothing more is handed out, and `pool remove` and `pool rekey` change both: each node restarts its cluster peer after its Kubo.
- **It needs no port.** Every peer listens on its own machine only. A worker's reaches the coordinator's through the coordinator's `--listen` port, as its Kubo does; only word of what is pinned and who is there travels that way. Workers' peers on different machines are not connected to each other, and need not be.
- **Each peer is its node**, with the node's ID and name, as its Kubo is. Its settings and its record of the pins are in `ipfs-cluster` in the node's data directory, and its log is `daemon.log` there.

`pool cluster` lists the members the coordinator's peer has heard from lately, and for each pin how many nodes hold all of it, out of how many were asked for, and which are still fetching or have failed:

```text
MEMBER                                                NAME   STATE
12D3KooWEixYEKkqKVZuwtPDXJaQs2Jevnv3CRz1B1sCsnnnWtSS  hub    answering
12D3KooWK5P7C6zFJ9ozcn9cHgQn1niHdbeW8YWH2cZJ5baCJqza  north  answering
12D3KooWRgaq6Nb65k1ZzAtBrLrwGS1X8GCGTnBXiupDuzZ3Z6eb  south  answering

each pin is to be held by 2 of them

CID                                                          COPIES  HELD BY     WAITING FOR
bafybeifnshe3y2i4sl46bbjbagfdu5xbsbq55uxqras5an73qgcqocpyeq  2/2     hub, north  -
```

**Sizing.** Every copy costs its size on the node that holds it, so a pool storing S with `--replicas N` needs (N + 1) × S of disk in all, S of it on the coordinator. The cluster allocates by what each Kubo reports as free, which is its `Datastore.StorageMax`, 10 GB unless changed, less what its repository holds. To offer more or less on a node, set it and restart the node:

```sh
IPFS_PATH=$(bin/sisyphusd data-dir)/ipfs ipfs config Datastore.StorageMax 200GB
```

A cluster peer is one more process per node; with a handful of pins it used 60 to 75 MB of memory here.

**Adding a storage node.** Start a worker with `--kubo --cluster`. It appears in `pool cluster` within half a minute. Pins that had fewer holders than asked for get it as one; pins that already had enough are not moved onto it.

**Removing one.** Stop it, or `pool remove` it. Thirty seconds later the coordinator's peer starts moving what it held to other nodes, if there are any that do not hold it already; `pool cluster` shows `1/2` until they have fetched it. A removed node is shut out by the change of key, and keeps on its disk what it had.

**Next to the built-in replication.** `--replicas` and `--replica-dir`, described under [Copies on other nodes](#copies-on-other-nodes), are replication done by `sisyphusd` itself, with no other program to run. The two are alternatives: this one is for pools that already run Kubo and want IPFS Cluster's allocation by free space and its repair, at the price of a second program on every node. `--replicas` means the same for both, copies besides the coordinator's own, and with `--cluster` it is the cluster that keeps them: the built-in scheme is then off, and a worker's `--replica-dir` holds nothing.

What to know before relying on it:

- Everything said of `--kubo` above and under [Known limits](#known-limits) holds here too: a node killed outright leaves both programs running, and both must be stopped by hand before it will start again.
- While the coordinator is down the copies stay where they are, and nothing is moved or repaired.
- Data is safe against the loss of a node only once `pool cluster` shows it held by more than one. A pin made a moment ago is still being fetched.
- Losing the coordinator's data directory loses the list of what is pinned, on the node and on the cluster, though the other holders still have the blocks. There is no way yet to rebuild a coordinator from them.
- A worker's disk is freed of what the cluster has let go only when its cache is next trimmed, which happens only with `--max-cache-bytes`. That limit counts what the cluster has the node hold, so set it well above that.
- A node that works for other pools (see [Trust](#trust)) joins their private networks and not their clusters.
- A pin put on the cluster by hand, with `ipfs-cluster-ctl` and under any name but `sisyphus`, is left alone and not shown by `pool cluster`.
- Tried with IPFS Cluster 1.1.6 and Kubo 0.43.1, on one machine: three nodes, a 50 MB file copied to a second node in four seconds, and to a third forty seconds after the second was stopped. Not tried across machines, nor with more than three nodes or more than a few pins.

### One port

To take part in a pool a node needs to make outgoing connections to its coordinator and nothing else. It opens no port, so it works from behind a home router or a firewall as it is. The coordinator is the one node that has to be reachable, on `--listen`.

That one port carries everything:

- **The node's own protocol**: jobs, tasks, uploads and downloads, over TLS.
- **A tunnel to the coordinator's Kubo**, for pools that use `--kubo`, inside that same TLS connection, and another to its cluster peer for pools that use `--cluster`.
- **libp2p**, by which the pool's nodes reach each other. The coordinator tells the two kinds of caller apart by how they begin.

**Workers reach each other through the coordinator.** Every node runs a libp2p host under its own ID. A worker keeps a place on the coordinator, which relays: when one worker wants a blob another has cached, it asks for that worker by ID, and the coordinator joins the two. Having met, the two try each other's own addresses, and if either can reach the other, on the same network for instance, they connect directly and the coordinator carries nothing more. Only the pool's members are let in or relayed for.

Opening more is optional, and buys speed:

| What a node opens | What it gets |
| --- | --- |
| Nothing (any worker) | Takes tasks. Fetches from the coordinator, and from other workers through it or directly where the network allows. With `--kubo`, reaches the coordinator's Kubo through the tunnel. |
| `--serve <addr>` on a worker | Other workers fetch from it at that address without the coordinator's help. |
| `--relay <addr>` on a worker | It becomes a full node: other members connect to its libp2p host there, and it relays between them as the coordinator does. |
| `--swarm-port <port>` on a coordinator with `--kubo` | Workers' Kubo daemons connect to the coordinator's directly rather than through the tunnel. |

A worker's Kubo tries the coordinator's open port first, if it has one, and settles for the tunnel if nothing answers within two seconds, so opening the port and then forgetting the firewall rule costs speed, not the pool. The worker logs which it chose.

What the tunnel costs: every byte between a worker's Kubo and the coordinator's passes through both `sisyphusd` processes and is encrypted a second time. On one machine it moves about 300 MB/s against 2 GB/s for a direct connection, so it matters on a fast network with fast disks and not otherwise. Run `go test -run xxx -bench . ./apps/sisyphusd/tunnel` to measure it on yours.

**Full nodes.** A worker started with `--relay <host:port>`, on a port the other members can reach, relays between them. It tells the coordinator where it is; every worker asks the coordinator from time to time which members relay and keeps a place on each, so any two can be joined through the coordinator or through any full node. Each full node counts what it carries and reports it to the coordinator, and `sisyphusd nodes` shows it in the `RELAYED` column. That count is what a reward for relaying would be worked out from; nothing pays one yet.

**Both kinds of address.** A coordinator started with `--listen :7700` listens on every IPv4 and IPv6 address the machine has. One that names an address listens on that one.

What is not there yet:

- **Two workers on different private networks may stay on a relay.** libp2p can sometimes connect such a pair directly ("hole punching"), and it is switched on. On one machine, relayed pairs do go direct by themselves. Whether they do across two real home routers has not been tried, so do not count on it. Step 4 of [`docs/multi-machine-test.md`](../../docs/multi-machine-test.md) says how to tell.
- **Which relay carries a connection is not chosen.** A node asks through all of them at once and uses whichever answers first, so a full node takes load off the coordinator without being preferred to it.
- **A full node's count is its own word.** Nothing checks it against what the nodes at either end saw.
- **Kubo daemons do not use the relay.** Two workers' Kubo daemons with no route between them exchange blocks through the coordinator's.

### Private jobs

A pool's storage is shared: any member can fetch any blob it can name, and with Kubo, whatever is on the pool's private network. A private job keeps its data from everyone but you, the coordinator, and the workers that run it.

```sh
bin/sisyphusd key new job.key                                   # make a key; keep the file safe
cid=$(bin/sisyphusd blob put --key-file job.key big.txt)        # sealed before it leaves this machine
bin/sisyphusd job submit --workload wordcount --key-file job.key --params "{\"input\":\"$cid\"}"
bin/sisyphusd blob get --key-file job.key <output CID>          # unsealed after it arrives
```

- **What is sealed.** The input you seal, everything the job's tasks store, and its stored result, so stores, caches and the private network hold only ciphertext. Nothing is sealed with your key itself: each stored file is sealed with a key of its own derived from yours, and what a job stores with a key derived for that job. Your key opens all of them.
- **Who gets which key.** The coordinator gets your key, when you submit the job. A worker assigned one of the job's tasks gets less: the job's own key, and the key of each sealed file the job's parameters name, wherever in them. With those it reads this job's inputs and what this job stored, and nothing else your key seals: not your other files, and not what your other jobs stored. Those workers still see this job's data in the clear, as they must to compute on it. No key is reported back in the job or logged.
- **A file a task reaches only through another file** is not one the job names, so its workers are given no key to it. Name every sealed input in the parameters.
- **Files sealed by an earlier version** were sealed with your key itself, and only that opens them. A job that names one runs, with its workers given your key whole as they once always were, and says so in its events. Store the file again and it gets a key of its own.
- **A worker from an earlier version is given no private job's tasks.** It would not know the new keys and would store unsealed what it was to seal, so a private job waits for a worker that is up to date.
- **Where the key is kept.** In the coordinator's database, readable by its owner only, and only until the job finishes: an unfinished job could not be taken up after a restart without it. When the job finishes the key is erased from the database's files. That is as much as software can promise; a disk may keep traces of what it once held.
- **What still shows.** The size of each blob, and that the job happened. The job's parameters, the small result it reports (for `wordcount`, the counts of words and of distinct words), what each task returns to the coordinator and error messages are not sealed; they travel over the pool's encrypted connections but are visible to clients of the coordinator. What a private job's tasks log is not passed on at all: a worker sends one line saying so in its place, since a log is kept unsealed and a `container` task's log is everything its command printed. An error may still quote the last line a failed command printed.
- **The same file is sealed the same way each time**, so storing it twice keeps one copy, and the workers of one job, who share its key, store the same blobs for the same work, which is how a private job is [verified](#verifying-results). Two jobs have different keys: run the same job twice and the results hold the same data under different CIDs, and nobody can tell from the store that they are alike. Within one job, and between two files of more than 4 MiB that begin with the same 4 MiB, which share a key, someone who can see two sealed blobs can tell whether they, or same-sized pieces at the same position in them, are identical.
- **Its record leaves the unsealed parts out.** A finished job's [record](#a-jobs-record) goes into the shared store, so a private job's holds no parameters, results or error messages.
- **Lose the key and the data is gone.** `key new` will not overwrite a key file for that reason.
- **A job with a key can still read unsealed inputs**, so public data and private can be mixed.
- **A job without a key is refused sealed data.** It has no key to it, and would otherwise compute on the sealed bytes as though they were the data. It is told the blob is sealed and to submit the job as private: when it is submitted, if the workload reads its input to split it, and as a failed task if not.

The encryption is put together from standard parts (AES-256-GCM, HMAC-SHA256, HKDF) but the arrangement is this project's own and has not been reviewed by a cryptographer. Treat it as keeping honest pool members out, not as proof against a determined attacker, until it has been.

### How long data is kept

A node keeps a blob for as long as something pins it, and deletes what nothing pins.

| Pin held for | Placed by | Lasts |
| --- | --- | --- |
| `user` | `blob put` and `blob pin`, by the node's owner | Until `blob unpin`, or for `--ttl` |
| `user:<node ID>` | `blob put` and `blob pin`, by a client of the pool | Until that client's `blob unpin`, or the owner's, or for `--ttl`. A client sees its own pins in `blob pins` and no one else's |
| `restored` | A coordinator that lost its store, on what it took back from its followers | Until the `blob unpin` of the node's owner, not of a client, or until the time the lost store kept it to; see [Copies on other nodes](#copies-on-other-nodes) |
| `job:<id>` | The coordinator, on a job's inputs and results | While the job runs, then for `--retain` (default 7 days) |
| `record:<id>` | The coordinator, on a finished job's [record](#a-jobs-record) | Until the job is forgotten (`--keep-jobs`) |
| `chat:<id>` | The node, on a file attached to a conversation with its planner | Until the conversation is deleted. The change is queued in the node's database with the conversation and applied to the store after, so one that fails, or that a stop cut short, is tried again every two seconds and when the node next starts. A pin of a file the node no longer holds can never be made, so it is reported to whoever attached it and dropped, and does not hold up the changes behind it |
| `chat-draft` | New private chat uploads through `StoreFile` with `chat_attachment` | 24 hours from the upload; another upload of the same content renews this grace period. Sending the message adds an independent `chat:<id>` pin |
| `recent` | Every upload | One hour, so there is time to pin it properly |
| `pool:<store>` | A storage follower, in its own store, on what its coordinator has it hold | Until the coordinator stops listing it; see [Copies on other nodes](#copies-on-other-nodes) |
| `pinning-service:<request>` | A request made through the [pinning service](#pinning-services) | Until the request is removed |

Blobs that only pass between a job's tasks are released as soon as the job ends. Several pins can hold one blob; it goes when the last has lapsed.

New chat attachments do not receive the permanent `user` pin used by ordinary file uploads. Deleting a conversation releases only that conversation's ownership; another conversation, a job or an explicit user pin can still keep the same content. Abandoned uploads, including completed uploads from a partially failed send, become eligible for collection after the 24-hour draft grace period. Deletion is not immediate: collection must run and every other pin must have expired or been released. Disabling automatic collection with `--gc-interval 0` requires manual `blob gc`.

The desktop checks both `chat-file-references-v1` and `chat-attachment-retention-v1` before uploading chat attachments. Older daemons cannot safely ignore the new upload flag. Existing permanently pinned uploads are not retroactively unpinned. The file list hides content that the store no longer holds, and reports storage failures rather than treating them as absence. Listing does not delete catalogue metadata or ownership: a concurrent upload or a later restore can make the content available again. Metadata compaction and paginated listing remain separate follow-ups.

```sh
bin/sisyphusd blob pins             # what is being kept, for whom, until when
bin/sisyphusd blob pin --ttl 72h <cid>
bin/sisyphusd blob unpin <cid>
bin/sisyphusd blob gc               # delete what nothing is keeping, now
```

A coordinator also collects every `--gc-interval` (default one hour), and with `--max-store-bytes` refuses uploads once its store reaches that size.

A job's result must be stored by the workload's aggregation step to be kept; a blob a task stored is treated as intermediate unless the aggregation stores it again.

### Fetching a file with a browser

A node can serve what it holds over plain HTTP, read-only, by content ID, as any IPFS gateway does:

```sh
bin/sisyphusd run --gateway-listen 127.0.0.1:8080
curl -H "Authorization: Bearer $(cat "$(bin/sisyphusd data-dir)/api.token")" http://127.0.0.1:8080/ipfs/<cid>
```

- **It is its owner's unless opened.** A request must show the node's API token, as a bearer token or as `?token=` in the address. Started with `--gateway-open`, anyone who can reach the address and knows a file's content ID can fetch it, which is how a link to a result is shared; a content ID cannot be guessed, but anyone given one can pass it on.
- **It speaks plain HTTP.** At a loopback address that is this machine talking to itself. At an address other machines can reach, the token crosses the network unencrypted with each request, and the node says so in its log when it starts: put a proxy that speaks TLS in front of it, or open it with `--gateway-open` and serve only what may be public. A token in `?token=` also ends up in browser histories and proxy logs.
- **A sealed file is never served**, with or without the token: what the gateway holds of it is what was sealed. Private data is fetched through the node, by whoever has its key.
- `?filename=report.pdf` names the file for the browser and decides what kind it is taken for; without it the kind is told from how the file begins. Part of a file may be asked for, so video can be played and seeked from it.
- It serves single files only. A directory, or a path inside one, is not something a node stores.

### Names for what changes

A content ID names bytes that never change. A node's **name** is its ID, and stands for whichever file the node last pointed it at:

```sh
CID=$(bin/sisyphusd blob put report.txt)
bin/sisyphusd name publish "$CID"       # prints the name: this node's ID
bin/sisyphusd name resolve <node-id>    # prints the content ID the name now stands for
curl -H "Authorization: Bearer $(cat "$(bin/sisyphusd data-dir)/api.token")" http://127.0.0.1:8080/ipns/<node-id>
```

- **What is published is a signed record**: the content ID, a sequence number that only goes up, and the time until which the record is good, signed with the node's key. It is an IPNS record, the kind Kubo makes, so other IPFS programs can read and check it. `name publish` signs it on the machine it is run on, with the key in `--data-dir`, and hands it to the node at `--addr`, which keeps it in its database, across restarts, and answers with it from then on.
- **A node publishes under its own name only.** A coordinator takes a record from its owner's command line and from each node it has admitted, worker or client, and each record must be signed by the node that hands it over. So a worker's name is answered for by its coordinator once `sisyphusd name publish --addr <coordinator> <cid>` has been run on the worker's machine. A record that another node signed is refused, whoever signed it, and so is one no newer than the record held. A node removed from the pool is no longer answered for.
- **Whoever resolves a name checks the answer.** `name resolve` asks the node at `--addr`, which must be this machine's own node or one it has joined, then checks the record's signature against the name and refuses a record that has expired. The node asked is not taken at its word.
- **Through the gateway**, `/ipns/<node ID>` serves the file the name stands for, under the rules `/ipfs/<cid>` has: the token unless `--gateway-open`, and never a sealed file. The name may be spelled as the node prints it (`12D3KooW...`) or as IPFS programs do (`k51...`). Asked with `Accept: application/vnd.ipfs.ipns-record`, or with `?format=ipns-record`, the gateway returns the signed record itself instead, as the IPFS gateway specification describes, for a reader who would rather check it than trust the gateway. Neither answer is marked as keepable for good: a browser or cache may keep it for the record's `--ttl`, or for what is left of the record's life if that is less.
- **A record runs out.** It is good for `--lifetime`, 48 hours unless told otherwise. After that the gateway answers 404 and `name resolve` refuses it, until the node publishes again. Nothing republishes a record that was published by hand.
- **A node has one name.** Publishing again replaces what the name stood for. There is no way yet to have several names, or to unpublish one other than letting its record run out.
- **Naming a file does not keep it.** A name can point at a file the node does not hold, or stops holding; the gateway then answers 404. `blob put` pins what it stores, and `blob pin` pins anything else.
- **With `--kubo`**, a coordinator also gives each record it takes to its Kubo, which puts it on the pool's private network, so that `ipfs name resolve /ipns/<node ID>` and `ipfs cat /ipns/<node ID>` work against any member's Kubo. That happens when a record is published and not again: Kubo lets go of a record it has not been given afresh for 48 hours, and a member's Kubo that is asked later than that will not find it. If Kubo will not take a record the node says so in its log and holds the record all the same.
- Names are slow to change by design: whoever has an answer may go on using it for the TTL. They are not for anything that must be seen to change within seconds.
- Not built: DNSLink (a domain name standing for a name), publishing to the public IPFS network, and telling readers of a change as it happens (IPNS over pubsub).

#### A coordinator's description of itself

Started with `--publish-record`, a coordinator keeps a small JSON document about itself stored, and its name pointed at it:

```json
{
  "id": "12D3KooW...",
  "version": "dev",
  "workloads": ["chat", "container", "..."],
  "members": [
    {"id": "12D3KooW...", "role": "worker"}
  ]
}
```

- **What it reveals** is exactly that: the node's ID, the version of `sisyphusd` it runs, the names of the workloads it takes jobs for, and the ID and role of every node admitted to its pool. It does not hold addresses, the labels given with `--name`, hardware, which members are connected, or anything about jobs.
- **Who can read it** is whoever can resolve the name and fetch the file: every member of the pool; anyone with the node's API token, through its gateway; anyone at all who can reach the gateway, if it was started with `--gateway-open`; and, with `--kubo`, every member's Kubo. A list of members is its owner's to give out, which is why this is off unless asked for.
- The node looks every 30 seconds for a change, and publishes a new record when the document differs. Each record is good for 48 hours and is replaced after 24. The document is pinned for as long as a record points at it.
- It uses the node's one name. Anything else published under that name is replaced by the description within 30 seconds.
- Only a coordinator has it. A worker-only node publishes no description of itself.

### In a bucket

A coordinator can keep the pool's stored data in a bucket of an object store that speaks S3, instead of on its own disk: Ceph, MinIO, SeaweedFS, Storj, Wasabi, AWS.

```sh
printf '%s\n%s\n' "$ACCESS_KEY" "$SECRET_KEY" > s3.keys && chmod 600 s3.keys
bin/sisyphusd run --s3-endpoint https://s3.eu-central-1.wasabisys.com --s3-bucket my-pool --s3-credentials s3.keys
```

- **The bucket must exist**; the node checks that it can reach it and stops if not. `--s3-prefix` puts the pool's objects under a name of their own, to share a bucket; `--s3-region` is for stores that have regions.
- **Nothing else changes.** A file has the same CID wherever its bytes are, so jobs, workers and clients need know nothing of it. Workers still fetch from the coordinator and each other; only the coordinator talks to the bucket, and the credentials never leave it.
- **Each block is an object**, of up to 256 KiB, under `blocks/`. What is kept and for how long is still the node's to decide: pins are in its data directory, and `blob gc` deletes from the bucket what nothing is keeping.
- **It is slower than a disk** when the bucket is far away: reading a gigabyte is four thousand requests, one after another. Nothing is cached on the coordinator. Use it for durability and room, and a bucket near the coordinator.
- It cannot be combined with `--kubo`, which is another answer to where the data is kept.
- Tried against SeaweedFS's S3 gateway. The others named speak the same dialect and have not been tried.

### Copies on other nodes

A coordinator holds the only durable copy of its pool's stored data. With `--replicas` it has other members each keep a copy of everything it has pinned, so that the data outlives the disk it is on. Those members are storage followers: worker-only nodes started with `--replica-dir`. There is nothing else to run.

```sh
# The coordinator: two copies of everything besides its own
bin/sisyphusd run --name alpha --replicas 2

# A follower, on another machine: a worker that also holds copies
bin/sisyphusd run --role worker --coordinator <addr> --join <invitation> \
    --replica-dir /mnt/big/sisyphus-copies

bin/sisyphusd blob replicas         # each pinned blob: copies held against copies wanted, and by whom
bin/sisyphusd blob replicas <cid>
```

- **What is copied** is every blob the coordinator has pinned for longer than the hour a new upload gets: what `blob put` and `blob pin` keep, and what jobs keep. When the last such pin on a blob is released or lapses, the followers delete their copies. The record of a job is not copied: it is linked data and not a file, and can be made again from the job.
- **A follower decides nothing.** Once a minute it asks the coordinator what to hold and says what it holds. It fetches what it lacks from the coordinator, checked against its CID, and deletes what is no longer listed. No other node can hand it anything to keep, and nothing it says changes what the coordinator keeps.
- **The coordinator signs each list**, and the follower keeps the last one beside its copies. That is what lets a coordinator that has lost its store take its data back without trusting anyone; see [below](#when-the-coordinators-store-is-lost).
- **The copies are in a store of their own**, in `--replica-dir`, written through to the disk like a coordinator's and apart from the worker's cache. Use the directory for nothing else: whatever is in it that the coordinator has not listed is deleted. It is always on the follower's disk, whether or not the pool uses `--kubo` or a bucket.
- **Which followers hold a blob** is settled by hashing each follower's ID with the blob's, so every blob ranks the followers in an order of its own and the first few hold it. Data therefore spreads evenly, and a follower that joins takes a share from each of the others while nothing moves between them.
- **A copy that is to move stays where it is** until the follower taking it over says it has it. Changing the followers never leaves fewer copies than there were.
- **A follower that is not heard from for ten minutes** is given up on, and the others are told to hold its share. One that is back sooner, after a restart say, causes nothing to move. Its copies stay on its disk while it is away.
- **A coordinator that has just started** waits the same ten minutes before handing blobs it already had to followers that lack them. Followers that held copies before the restart say so in that time, and nothing is copied twice. Blobs pinned after the start are handed out at once.
- **Workers can fetch from a follower** what it holds for the pool, as they fetch from each other.

#### How many followers

`--replicas N` asks for N copies besides the coordinator's own. The data then survives the loss of any N nodes.

| To survive losing | Run | Each follower holds |
| --- | --- | --- |
| Any one node | `--replicas 1` and one follower, or more | All of it, with one; half, with two |
| Any one node, with no wait for copies to be made again | `--replicas 2` and three followers | Two thirds |
| Any two nodes | `--replicas 2` and two followers, or more | All of it, with two |

- With F followers, each holds about N/F of what is pinned. Give each room for N/(F-1) of it, which is what it holds once another has been lost.
- With fewer followers than N, each blob is held by all there are, and `blob replicas` says how many are missing.
- **Adding a follower:** start it. It is given its share within a minute and fetches it; the others delete what it took over once it has it.
- **Removing one:** stop it. After ten minutes the others are told to hold its share, for which they need the room. `pool remove` then stops it coming back, and its directory can be deleted.
- A follower has no limit of its own on the disk it uses. The coordinator's `--max-store-bytes` bounds what there is to copy.

#### When the coordinator's store is lost

Start the coordinator again with the same key. It takes back what its followers hold, by itself.

```sh
bin/sisyphusd run --name alpha --replicas 2    # the same data directory, with node.key in it and a new, empty blobs/
bin/sisyphusd blob replicas                    # says what has come back, and what has yet to
bin/sisyphusd blob pins                        # what came back is held for "restored"
```

- **The coordinator signs what it tells each follower to hold.** Each answer to a follower is a list: the blobs, until when each is kept, the name of the store, and a number that only goes up, signed with the node's key. The follower keeps the last list in `kept.list` in its `--replica-dir`, beside the copies.
- **A store has a name**, kept beside its pins. A coordinator that comes back with its key and an empty store, because the disk was replaced, has a new name. A follower that holds copies for the old one is not told to delete them. It is asked for its list.
- **The coordinator checks its own signature**, and takes the follower's word for nothing. If the list is its own, it fetches each blob named from that follower, checks it against its CID, and pins it. A list that was changed, or that another key signed, is ignored and logged, and nothing is taken back from it.
- **What comes back is pinned for `restored`**, until the time the list gave: the latest of the pins the lost store had on the blob. What had no expiry has none. What has since run out is not taken back, and the follower deletes it. Who had pinned each blob is not in the list. `blob unpin` releases a restored blob as it does one pinned by hand.
- **Several followers each show a list.** Each is taken. A blob that two lists name is kept until the later of the two times.
- **It starts when the follower next asks**, within a minute of the follower finding the coordinator again, and is done when everything named has been fetched. What cannot be fetched is tried again each minute. `blob replicas` says how many blobs have come back and how many are still to come.
- **A blob that is asked for** before then is fetched back from a follower there and then, as it always was.
- To give the old data up instead, stop each follower and empty its `--replica-dir`.

##### When `blob restore` is needed

`sisyphusd blob restore` fetches back everything the followers hold from an earlier store, on the followers' word, and pins it for `user` until unpinned. With the key, it is not needed. It is for what the coordinator cannot check:

- **The key was lost too.** A new key makes a new node, and a list signed with the old key cannot be told from a forgery. Start the new coordinator with `--formerly <the old node ID>`, so that it keeps what followers hold for the old node instead of having them delete it. On each follower's machine run `sisyphusd pool join --addr <coordinator> <invitation>` with an invitation from the new coordinator, and start the follower again with the same `--replica-dir`. Then run `blob restore`. The old ID is in `pins.json` in each follower's `--replica-dir`: every pin there is owned by `pool:<the old node ID>/<the old store's name>`.
- **A follower has no list**: it last heard from a coordinator older than signed lists, or its `kept.list` is gone or damaged.
- **A follower holds blobs its list does not name.** While a follower is still holding for an earlier store, because something of it has yet to come back, it is given no new list, and what it is handed in that time is on none. If the store is lost again before that is over, those blobs are kept on the follower and not taken back.

`blob replicas` says which case applies: blobs "that it has yet to take back" need nothing, and blobs "that no list signed by this node names" need `blob restore`. Run when there is nothing of the kind, it says so and does nothing.

##### What a follower can still do

- **It cannot add to what is kept.** Only what the coordinator pinned is ever in a list it signed. Blobs from an earlier store that no signed list names are never in one.
- **It can show an old list.** A follower that kept a list from before a pin was released can show it to a coordinator that has lost its store, and the blob comes back, until the expiry that list gave it, or until unpinned if it had none. That brings back only what the coordinator once chose to keep, and it needs the coordinator to have lost its store: a list of the store it has now brings nothing back.
- **A blob taken back and then released stays released** while the coordinator runs. A coordinator that is started again while a follower still shows a list of the lost store takes it back once more.

#### What this does not protect

- **The coordinator's database and key.** `node.db` holds the jobs, the members and the invitations, and `node.key` is who the node is: followers know their coordinator by it, and it is what makes a list the coordinator's own. Neither is copied anywhere. Back them up. Without the key, getting the data back takes `--formerly`, a new invitation for every follower, and `blob restore`, and the expiries are lost.
- **Who pinned what.** Followers hold blobs and, in the list, until when each is kept. A blob a job was holding while it ran comes back with no expiry, and stays until `blob unpin`.
- **What is not pinned**, and what was pinned in the last minute or so, before a follower had asked and fetched it.
- **A follower is asked to show what it holds.** Each time it says what it holds, about once a minute, the coordinator picks up to four of those blobs and asks for the digest of a random stretch of each, up to 64 KiB, under a number made up for the occasion. Only something with those bytes to hand can answer, and it has thirty seconds. A copy that is not shown is not counted, so another follower is told to hold the blob, and the follower that failed is asked about it first the next time. `blob replicas` has a SHOWN column, and says how often a follower has failed. Blobs never shown come before those shown, and those shown longest ago before the rest, so every copy is asked about in turn.
- **What that does and does not show.** That the follower could produce those bytes of that blob in that time. Not that it holds the rest, though it cannot know which bytes will be asked; not that it will hold them tomorrow; not that it would hand them over; not that two followers are two disks. A follower that keeps nothing and fetches a blob from someone else when asked passes if it is quick. And a follower from before this check is asked nothing and is still taken at its word. It is a check against a disk that lost data and a follower that lies cheaply, not a proof of storage in the sense a paid network needs: see [the research note](../../docs/research/storage-and-assets.md).
- **A follower can read what it holds**, as any worker can fetch any blob. Private jobs seal their data, and the copies are of the sealed form.
- There is one number of copies for the whole pool, not one per pin, and copies are whole: no erasure coding.
- The intervals are fixed: a minute between asks, ten minutes before a follower is given up on.
- The list of what to hold travels whole each time, which bounds a pool at about sixty thousand pinned blobs. A follower that shows its list sends it along with what it holds, so a follower that holds more than about thirty thousand blobs of a lost store cannot show its list: it keeps what it has, and `blob restore` takes it back.
- A coordinator takes a list back while it answers the follower that showed it. That follower hears nothing new until it is done.

### Pinning services

The [IPFS Pinning Service API](https://ipfs.github.io/pinning-services-api-spec/) is how one machine asks another, over HTTP, to keep data by its CID. A node speaks it both ways: it can be the service, and it can ask one.

**A node as the service.** A coordinator started with `--pinning-listen` serves the API, so that the `ipfs` program, or anything else that speaks it, can have the node keep data:

```sh
bin/sisyphusd run --pinning-listen 127.0.0.1:7710
ipfs pin remote service add pool http://127.0.0.1:7710 "$(cat "$(bin/sisyphusd data-dir)/pinning.token")"
ipfs pin remote add --service=pool --name=results.tar <cid>
ipfs pin remote ls --service=pool
ipfs pin remote rm --service=pool --cid=<cid>
```

- **It has a key of its own**, in `pinning.token` in the node's data directory, made the first time the service starts. It is not the node's API token. That one controls the node; this one is handed to other programs and other machines, and lets its holder do three things: ask the node to keep data, list the requests made through this API, and remove them. To change it, delete the file and restart the node.
- **What the node holds is pinned at once.** The pin is held for `pinning-service:<request ID>` and shows in `blob pins`. It lasts until the request is removed, and then garbage collection takes the data if nothing else keeps it. Two requests for the same data are two pins.
- **What the node does not hold, it can fetch only with `--kubo`.** Its Kubo connects to the `origins` the request names and fetches the data, and the request goes from `queued` through `pinning` to `pinned`, or to `failed` if a block cannot be found within thirty seconds. Four requests are fetched at a time. Fetches left unfinished when the node stops are taken up when it starts again. Without `--kubo` the node has nothing to fetch with, and such a request is `failed` at once, with the reason in its `info`.
- **A node's Kubo fetches only from the pool's private network.** It can connect to a peer only if that peer holds the pool's swarm key. An `ipfs` daemon on the public IPFS network can ask this service to pin something, and will be answered, but cannot supply the data: the request fails unless the node already holds it. A daemon that is to supply data must be given the pool's `swarm.key`, which also lets it read everything on the pool's network that is not sealed.
- **It keeps files and directories**, which are raw and dag-pb blocks. A request for any other kind of IPLD data is refused.
- **Listing** takes the API's filters: `cid`, `name` with `match`, `status`, `before`, `after`, `limit` and `meta`. Replacing a request, with `POST /pins/<request ID>`, keeps the old data pinned until the new data is here.
- **The only limit is the store's.** With `--max-store-bytes`, a request that needs fetching is refused once the store is that full. The check is made before a fetch, not during it, so one fetch can take the store past the limit. There is one key, and no accounting of who asked for what.
- **It is plain HTTP.** The key crosses the network unencrypted. Listen on this machine only or on a network you trust, or put a server that does TLS in front of it.
- `ipfs pin remote add` says only that a failed request failed. The reason is in the request's `info`, which `blob pins-remote` shows.

**A node asking a service.** `blob pin-remote` asks a pinning service to keep a blob as well: another node, an IPFS Cluster of your own, or a commercial service.

```sh
bin/sisyphusd blob pin-remote --service https://pins.example.com/psa --key-file service.key --wait <cid> results.tar
bin/sisyphusd blob pins-remote --service https://pins.example.com/psa --key-file service.key
bin/sisyphusd blob unpin-remote --service https://pins.example.com/psa --key-file service.key <request ID>
```

- `pin-remote` prints the request's ID and its status. With `--wait` it asks every two seconds until the service says `pinned` or `failed`, and prints that. A failed request ends the command with an error and the service's reason.
- **What the service is told:** the blob's CID; the name, if you give one; and where the pool's Kubo daemons can be reached, as the request's `origins`, so that the service knows where to fetch from. Those are the IP addresses, ports and node IDs of the coordinator's Kubo, if it was started with `--swarm-port`, and of the Kubo daemons connected to it. Like any server, it also sees the address the request came from. `--no-origins` leaves the addresses out. A node without `--kubo` has none to give.
- **The data is not uploaded.** The service has to fetch it, and a service on the public IPFS network cannot fetch from a pool: the pool's Kubo daemons answer only holders of its swarm key. Such a service will take the request and then fail it, or leave it waiting. For it to succeed, either the data has to be public in some other way, such as added to a Kubo on the public network, or the service has to be on the pool's network.
- **A service on the pool's network** is one whose IPFS daemon holds the pool's `swarm.key` and can reach the pool's Kubo daemons. Another coordinator becomes one if a copy of the key is put in its data directory as `swarm.key` before it is started with `--kubo --pinning-listen`; the asking coordinator needs `--swarm-port` so that its Kubo can be reached. The key is a secret: whoever holds it can read everything on the pool's network that is not sealed. `pool rekey` and `pool remove` change it, and a service outside the pool then has to be given the new one by hand.
- The request is sent from the machine the command runs on. The key is read from a file so that it appears in no command line, and it is not given to the node.
- `pins-remote` lists every request the service has for the key, whoever made it.
- A sealed blob is kept as it is stored, sealed. The service never has its key.
- Tried between two nodes, and with the `ipfs` program as the client of a node. Not tried against a commercial service or IPFS Cluster.

## What each machine has

A worker looks at the machine it runs on and tells its coordinator what it finds: the processor, how much memory, and any NVIDIA graphics cards. `sisyphusd nodes` shows it.

```sh
bin/sisyphusd run --offer-memory-mb 16384 --offer-gpus 1 ...     # offer the pool less than the machine has
bin/sisyphusd job submit --min-memory-mb 8192 --gpus 1 ...       # a job that needs that much
```

- **A job can ask for memory and graphics cards**, and its tasks then go only to workers that have them. Asked to split itself, it splits to fit the workers that qualify. If none does, it waits until one connects.
- **The owner decides what to offer.** `--offer-memory-mb` and `--offer-gpus` cap what the node says it has. They do not hold a workload to it: nothing yet stops a task using more memory than its node offered.
- **What is found, and where.** Processor and memory are read from `/proc` on Linux, asked of `sysctl` on macOS and of PowerShell on Windows. Where none of those answers they are left unknown, and such a worker is given no job that asks for memory unless its owner says how much to offer. Graphics cards are found wherever NVIDIA's `nvidia-smi` is installed; other makes are not looked for.
- **A requirement is a count, not a choice.** A task that asks for a graphics card is sent to a machine with one. Which card it then uses, and keeping two tasks off the same one, is the workload's business for now.

## The planner

A node that coordinates a pool can be asked questions in plain words. A language model decides what needs computing, has the pool compute it, reads the result, and answers.

```sh
bin/sisyphusd model set --model qwen3:8b                         # a model served by Ollama on this machine
bin/sisyphusd model set --provider anthropic --model claude-opus-5-5 --api-key-file key.txt   # or Claude
bin/sisyphusd run --api-listen 127.0.0.1:50051                   # the node, with its local API on
bin/sisyphusd ask "How many primes are there below ten million?"
bin/sisyphusd ask --chat <id> "And below a hundred million?"     # carry the conversation on
```

- **Which model.** `model set` takes `--provider ollama` (the default), `--provider anthropic` for Claude, or `--provider openai`, which is for OpenAI and for anything that speaks as it does: most hosted services and local servers. `--url` says where the service is if not in its usual place, and `--api-key-file` gives a key if it wants one. `model list` asks the service what it offers, and marks any model that cannot call tools and so cannot plan; `model show` prints what is set, without the key; `model providers` lists the kinds of service. `model pull --model llama3.1:8b` fetches a model into Ollama and `model remove` deletes one, each taking `--url` for an Ollama somewhere else. A running node picks up a change at the next question.
- **Claude.** With `--provider anthropic` and no key file, the node uses the key Anthropic's own tools are set up with: `ANTHROPIC_API_KEY`, or a profile from `ant auth login`. Claude's replies are kept whole, with its reasoning, and handed back to it on the next question, as Anthropic asks; if the pool's workloads have changed since, so that Anthropic no longer accepts the reasoning, the question is asked again without it.
- **What the model can do.** Two things: run a job on the pool and read the result, and look up a job run earlier. It is told which workloads the pool has and what parameters each takes. It has no other reach: it cannot run commands, read files or touch the node's settings.
- **What you see.** `ask` prints the model's words as they come, each job it starts, and what each returned. The jobs are ordinary jobs: `job get` and `job logs` work on them.
- **Conversations are kept** in the node's database, with every request the model made and what came back, and are there for the desktop to list and read.
- **It needs the node's token**, like anything else that spends the pool's time. `ask` reads it from the data directory.

What to know before relying on it:

- **It is as good as the model.** A small model may ask for the wrong parameters, or answer without computing. The model must support tool calling; many small ones do not.
- **It has been tried with one real model**, `llama3.1:8b` from Ollama, which ran the right jobs and answered correctly over two turns. The tests use a stand-in. Anthropic's and OpenAI's services have not been tried for want of keys. Expect the instructions the model is given to need tuning as more models are tried.
- **The key is kept in the node's database**, which only its owner can read, not in the operating system's keychain.
- **One question at a time per conversation.** Two asked at once in the same conversation are recorded in whichever order they finish.
- **It gives up after eight rounds** of asking for more without answering.

## Jobs made of jobs

A `graph` job runs several jobs as one. Each step is a job of some workload; a step may use what an earlier one produced, and waits for it.

```sh
bin/sisyphusd job submit --workload graph --params '{"steps":[
  {"name":"count","workload":"wordcount","params":{"input":"<cid>"}},
  {"name":"again","workload":"wordcount","params":{"input":"${count.result.output}"}},
  {"name":"low","workload":"primes","params":{"from":0,"to":1000000},"tasks":2},
  {"name":"span","workload":"primes","params":{"from":0,"to":"${low.result.count}"},"after":["again"]}]}'
```

- **References.** In a step's parameters, `${name.result}` is an earlier step's result, `${name.result.field}` a field of it, `${name.outputs.0}` the first file it stored and `${name.job_id}` its job. A string that is nothing but one reference becomes the value itself, so a number stays a number.
- **Order.** A step waits for every step it refers to and any it names under `"after"`. Steps that wait for nothing run side by side. A graph in which a step waits, however indirectly, for itself is refused when submitted.
- **Each step is an ordinary job**, with `parent_job_id` and `step` saying what it is a step of. `job get` and `job logs` work on it, and its tasks go to workers as any job's do. The graph's own tasks are its steps, carried out by the coordinator; no worker needs to know of graphs.
- **Failure and stopping.** A step that fails, or whose job is cancelled, fails the graph, and steps not yet begun are not run. Cancelling the graph cancels the jobs its steps are running as.
- **A restart** of the coordinator takes a graph up where it was: steps that had finished are not run again, and one that was running goes on as the job it was.
- **A private graph** gives its key to every step.
- Up to 64 steps. There are no loops or conditions: a step runs or the graph fails.

## Thinking on the pool

A pool can serve language models as well as run jobs. A worker whose owner runs Ollama offers its models; the coordinator offers all of them at one address, speaking OpenAI's dialect, so that whatever can talk to such a service can think on the pool.

```sh
bin/sisyphusd run --role worker --coordinator <addr> --join <invitation> --models-from http://127.0.0.1:11434   # a worker with models
bin/sisyphusd run --inference-listen 127.0.0.1:11435                                                            # the coordinator

curl -H "Authorization: Bearer $(cat "$(bin/sisyphusd data-dir)/api.token")" http://127.0.0.1:11435/v1/models
bin/sisyphusd model set --provider openai --url http://127.0.0.1:11435/v1 --model llama3.1:8b \
    --api-key-file "$(bin/sisyphusd data-dir)/api.token"      # the node's own planner, thinking on its pool
```

- **Each reply is a job** of the `chat` workload, given to a worker that serves the model asked for, and shows among the pool's jobs like any other. The local API's `ListWorkers`, and an agent's `pool_status`, show which models each worker serves.
- **The service** answers `GET /v1/models`, `POST /v1/chat/completions`, with tool calls, and `POST /v1/embeddings`, for up to 128 texts in a request, which are shared out among the workers serving the model.
- **Many prompts at once** is the `prompts` workload: `job submit --workload prompts --params '{"model":"llama3.1:8b","prompts":["...","..."],"system":"..."}'`, or `"input"` with the CID of a stored file that has a prompt on each line. The prompts are shared out among the workers serving the model and the answers come back in order. This is where several machines with models are faster than one. Its key is the node's API token, and it listens on this machine only.
- **What a worker's models are doing.** `sisyphusd nodes` adds a line for each worker that serves models, such as `laptop serves llama3.1:8b (loaded, 75% on a graphics card), qwen3:8b`. A model marked loaded is in its server's memory and answers at once; one that is not is loaded when first asked for, which can take a minute. The share is how much of it fits on the machine's graphics cards: at 100% the card does all the work, and below that the rest is done by the processor and replies come slower. A model that does not fit is a reason to serve a smaller one. The worker asks its Ollama for this with each heartbeat; a server that is not an Ollama says nothing, and nothing is shown.
- **Which worker answers.** A reply that names a model without a worker goes to the worker with the fewest replies on that model in hand, since a model server answers about one at a time however many slots its worker has; among those, to the one with the model in memory and most of it on a graphics card; and then to the one with most slots free.
- **What the owner can limit.** `--slots` is how many tasks run at once, `--offer-memory-mb` and `--offer-gpus` are how much of the machine the pool is told about, and leaving out `--models-from` serves no models at all. With `--models-from`:
  - `--models-only` offers only the models it names (`--models-only llama3.1,qwen3:8b,gemma*`: a family, an exact model, a pattern), so that a large model kept on the machine for its owner's own use is not handed to the pool. A worker refuses a model it does not offer even if a job names it.
  - `--models-gpu-layers` is how many of a model's layers go on graphics cards for the pool's jobs: `0` keeps the pool's work off the card altogether, a number below the model's layer count shares the model between card and processor, and the default lets Ollama put on as many as fit.
  - `--models-threads` is how many of the processor's threads a model may use, and `--models-context` how many tokens of a conversation it holds (more takes more memory).
  - `--models-keep-alive` is how long a model stays loaded after it last answered for the pool: `0` frees its memory at once, `30m` keeps it ready.
  - These are put on every request the worker makes of its Ollama, over whatever a job asked. The worker speaks to Ollama in Ollama's own dialect for this, since its OpenAI-style service ignores such settings; jobs are still written and answered in OpenAI's.
  - `--offer-gpus 0` hides the card from jobs that ask for one; on its own it does not stop Ollama using the card for the models served. `--models-gpu-layers 0` does.
- **What only Ollama itself can be told.** Some limits belong to the Ollama server as a whole, not to a request, and are set where Ollama runs, as environment variables, followed by a restart of Ollama: `OLLAMA_MAX_LOADED_MODELS` (how many models it holds in memory at once), `OLLAMA_NUM_PARALLEL` (how many requests one model answers at once), `OLLAMA_GPU_OVERHEAD` (memory on each card it leaves alone), `OLLAMA_KEEP_ALIVE` (the keep-alive for requests that do not give one, its own owner's included), `CUDA_VISIBLE_DEVICES` or `ROCR_VISIBLE_DEVICES` (which cards it may see at all), and `OLLAMA_HOST` (where it listens). On Windows they are set in the system's environment variables and Ollama is quit from the tray and started again; on Linux, with `systemctl edit ollama`; on macOS, with `launchctl setenv`. See Ollama's FAQ for each.
- **`--models-from` is a decision about your machine.** Whoever may submit jobs to the pool can then use your models and the power they draw, and your machine sees every conversation it is given: there is no sealing of a conversation from the machine that has to read it.
- **Keep a conversation on a machine you trust** by asking for the model on a worker by name: `"model": "llama3.1:8b@rig"`. It goes to the worker named `rig` or, if that worker is not there, is refused; it never goes elsewhere. A name is whatever a worker calls itself, so in a pool with workers you do not run, write the worker's node ID in place of its name, `"model": "llama3.1:8b@12D3KooW…"`, as `sisyphusd nodes` shows it: the coordinator knows that from the key the worker connected with. Asking for a model on your own machine's worker keeps the conversation at home while still going through the pool.
- **A reply arrives as it is written.** A client that asks for a stream gets the words a few times a second, and at the end the tools the model asks for and what it used. The worker logs the reply in pieces as it comes, so `job logs` shows it too.
- **A model may be named loosely** at the service: in another case, without its tag when the pool has a `latest`, or by its family alone (`llama3.1`) when the pool serves one model of that family. A name that could mean two models is refused, with the list of those served. A `chat` job submitted directly names its model exactly.
- **A model fetched later is offered within seconds.** A worker tells the coordinator when its models change, and a conversation that was waiting for one is then had.
- **Conversations are kept** with the pool's other jobs, parameters and result, for as long as jobs are kept (`--keep-jobs`).
- **From another machine**, the models are reached through that machine's own membership of the pool, not by opening the service to the network. Join as a client (`pool join` with a client invitation), then `sisyphusd inference --addr <coordinator>` offers the pool's models at `http://127.0.0.1:11435/v1` to programs on that machine, with the key in `api.token` in its data directory. Requests travel as jobs over the connection the pool already authenticates. A machine that is in the pool only as a worker cannot use it this way: workers run the pool's work and do not submit any.
- **A worker lost part way through a streamed reply** ends the stream with an error saying so, and the job is stopped. Another worker would begin the reply again, and a reply whose beginning came twice would look right and be wrong. Ask again. A reply not asked for as a stream is simply made again.
- **A call a model writes out is still made.** A small model sometimes writes the tool call it means as text, `{"name": "run_job", "parameters": {...}}`, often with Python's `True`, `False` and `None`, rather than making it. When the whole of a reply is such an object and names one of the planner's tools, the planner takes it as the call. Text that opens the way such a call does is held back until it is known which it is, so the call is not shown as a line of JSON first; a reply that says anything besides is an answer, and is shown in full once that is plain, or once it has run past a few thousand characters. With `llama3.1:8b` through Ollama's OpenAI-style endpoint, three replies in six came this way and all three ran their job.
- It has been tried on one machine with `llama3.1:8b` on a processor: the planner, set as above, ran the right job and answered correctly, and a streamed reply arrived a word or two at a time. It has not been tried on a graphics card, or with workers on other machines.

## Agents

The planner is the node's own agent. Any other agent that speaks the Model Context Protocol, such as Claude Code, Claude Desktop or Cursor, can use the pool too, through `sisyphusd mcp`:

```sh
bin/sisyphusd run --api-listen 127.0.0.1:50051        # the node, with its local API on
claude mcp add sisyphus -- /path/to/sisyphusd mcp     # Claude Code; other agents take the same command in their own settings
```

```json
{ "mcpServers": { "sisyphus": { "command": "/path/to/sisyphusd", "args": ["mcp"] } } }
```

The second form is what Claude Desktop (`claude_desktop_config.json`), Cursor (`~/.cursor/mcp.json`) and most others take. The agent starts `sisyphusd mcp` itself and talks to it over standard input and output. It takes `--data-dir`, `--api` and `--addr` if the node's data directory, `--api-listen` and `--listen` are not the usual ones.

There are 44 tools: 19 that only look, 14 more that use the pool, and 11 more with `--admin`.

| Tool | What it does |
| --- | --- |
| `pool_status` | The workers connected, with their cores, memory, graphics cards, the models they serve and how busy they are. |
| `list_workloads` | What the pool can run, the parameters each workload takes, and how many connected workers run it. |
| `run_job` | Runs a job and waits for its result, or returns at once with `detach`. It takes what `job submit` takes: `tasks`, `mode`, `private`, `min_memory_mb`, `min_gpus`, `task_timeout_seconds`, `verify` and `verify_share` to have [workers check each other](#verifying-results), and `audit_share` to have [the node check them itself](#checked-by-the-coordinator-itself). |
| `get_job`, `list_jobs`, `cancel_job`, `job_logs`, `wait_for_job` | Look at a job, list them, stop one, read what it logged, wait for one to finish. |
| `store_file`, `fetch_file`, `list_files`, `remove_file`, `fetch_outputs`, `save_result` | Put a file from this machine in the pool's store, bring one back, list them, stop keeping one, bring back everything a job stored, write a job's whole result to a file. |
| `list_pins`, `pin_file`, `unpin_file` | What the node keeps, for whom and until when ([How long data is kept](#how-long-data-is-kept)); keep a file, such as a job's output, for good or for so many hours; release it. |
| `storage_status` | Which other nodes hold copies of what is pinned, against how many should: the storage followers of [Copies on other nodes](#copies-on-other-nodes), or the members of the pool's [IPFS Cluster](#ipfs-cluster-pins-held-by-several-nodes), whichever the pool uses, and it says so plainly of a pool that uses neither. |
| `get_job_record` | A finished job's [record](#a-jobs-record): its ID and its nodes as JSON. With `verify` it is checked against the job as `job record --verify` checks it, and with `check_commitments` a private job's commitments are checked with the node's own `private.key`. |
| `resolve_name`, `publish_name` | What a node's [name](#names-for-what-changes) stands for, checked against the name as `name resolve` checks it; point this node's name at a file. |
| `ask_model`, `compare_texts` | Has a model served by one of the pool's workers answer a prompt; says which texts mean most alike, by an embedding model the pool serves. |
| `ask_planner`, `list_chats`, `get_chat`, `delete_chat` | Hand a whole question to the node's own planner, with `files` the content IDs of stored files it is about, which the node then keeps for as long as the conversation; read or forget its conversations. |
| `list_peers`, `list_members`, `list_address_book` | The nodes this node knows of, with their countries and which way work flows; who is in its pool; which nodes it connects to each time it starts. |
| `get_model`, `list_models`, `list_model_providers` | What the planner plans with, what its service offers, and the kinds of service there are. |
| With `--admin`: `create_invitation`, `remove_member`, `join_pool`, `connect_peer`, `forget_peer`, `set_peer_trust`, `set_model`, `pull_model`, `remove_model`, `collect_garbage`, `restore_files` | Change the node itself: who is in its pool, which nodes it trusts and connects to, what it plans with; clear its store of what nothing pins now, and take back from its followers what a lost store held (`blob gc`, `blob restore`). |

Besides tools, the server gives an agent things to read (`sisyphus://pool`, `sisyphus://workloads`, and the skill itself as `sisyphus://skill/SKILL.md` and `sisyphus://skill/recipes.md`), three prompts (`run-on-pool`, `pool-report`, `keep-result`), and, while `run_job` or `wait_for_job` waits, word of each task as it finishes.

What the agent may do is yours to set, when you give its settings the command:

| Flag | What it does |
| --- | --- |
| `--files-under <dir>` | The only directory `store_file` may read from and `fetch_file` write to. It is the directory the server is started in unless you say otherwise; `/` is anywhere you may. A link inside it that leads outside is outside. |
| `--read-only` | Only the tools that look. Nothing is run, stored, written or changed. |
| `--images a,b` | The only container images `run_job` may run, in a `container` job or in any step of a `graph`. An image a step would take from an earlier step's result is not a listed one. `ask_planner` is refused while images are listed, since the planner chooses what to run itself. Any, if not given. |
| `--admin` | Adds the tools that change the node itself. Without it there are none. |

- **It is a client of the local API** and holds the node's token, which it reads from the data directory.
- **Pins, copies, names and records are not in the local API**, so for those tools it reaches the node's main address, `--addr`, as the command line does: with the node's key in `--data-dir`. That connection is made the first time one of them is called. If it cannot be made, or the node is not at `--addr`, those tools fail saying why and every other tool goes on working. The token is not sent there.
- **`publish_name` signs with the node's key**, as `name publish` does, and `get_job_record` with `check_commitments` sends the node its own `private.key`, as `job record --key-file` would. It reads that key and never makes one, and no tool hands either key to the agent.
- **Pinning services are not offered.** `blob pin-remote` and its fellows need another service's key, and an agent is given no key to pass on.
- **With no flags it can use the pool and not reconfigure it.** It can spend the pool's time, run container images on workers that allow containers, and read and write files in the directory it was started in. It will not write over a file that is there unless told to in so many words.
- **`--admin` is the node's keys.** An agent with it can invite anyone into the pool and join the node to others. It is never given a model service's key: `set_model` keeps the one already set.
- **A skill comes with it**, carried in the daemon. `sisyphusd skill install` puts it where Claude Code keeps skills (`~/.claude/skills`), `--dir` puts it elsewhere, and `sisyphusd skill show` prints it. It tells an agent how to use a pool well: look before submitting, how to split work with a container, how to have the pool's models think, what goes wrong; `recipes.md` beside it has worked shapes for common jobs. It works with the tools or, without them, with the command line. Its source is [`skills/sisyphus`](../../skills/sisyphus/SKILL.md).
- **It has been tried from Claude Code**, with the skill installed and the server kept to one directory: asked to describe the pool, upper-case a file across three tasks and have one of the pool's models answer a question, it read the skill, ran a `container` job and `ask_model`, and wrote the right file. A second run, on Opus, labelled eight reviews with one `prompts` job and found the two most alike with `embed`; what it stumbled on (a result of vectors too long to hand back) is why `save_result` and `compare_texts` exist. A third, with the tools for what the node keeps and no skill, counted a file's words, pinned the result for a day, verified the job's record, found from `storage_status` that no other node held a copy, and published and resolved the node's name. It had to find out from `list_pins` that the job's own pin outlasts a day's pin, which is why `pin_file` now answers with every pin on the file. A fourth, with `--admin` and the skill, against three workers: it made an invitation and saved it, saw the worker it admitted join, counted primes with half the tasks verified and said which task had been run once and so how far the count could be trusted, counted a private file's words with every task verified, checked that job's record and its eighteen commitments and said what the record hides, published the other job's record under the node's name, found that one disk held everything, and removed its input and collected garbage, explaining that nothing was freed while the job's own pin stood. Nothing failed. It read `list_members` leaving out the node itself as something to remark on, which the tool now says, and it said `pin_file` was not among the tools it had loaded, though the server offers it; that looks like Claude Code holding some of a long list of tools back until they are searched for, and has not been looked into. Other agents have not been tried.

## Following and stopping a job

```sh
bin/sisyphusd job logs <job-id>                 # what has happened to it, and what happens next, until it is over
bin/sisyphusd job cancel <job-id>               # stop it
bin/sisyphusd job submit --timeout 10m ...      # stop and retry any attempt at a task that runs longer
```

- **Events.** A coordinator records what happens to each job: submitted, each task started, succeeded, failed, lost with its worker or timed out, and how the job ended. A [verified](#verifying-results) job also has each result a worker returned, each time the results for a task differed, and each task that was verified after all because of a [spot check](#spot-checks). `job logs` prints them and follows a running job to its end. They are kept in the database with the job.
- **Logs.** A task can log lines as it runs, and those appear among the job's events with the task and worker they came from. The two built-in workloads log one line each.
- **Progress.** A task can say how far along it is. `job get` and the desktop's API give it for each task, from 0 to 1, and for the job as the mean of its tasks. It is as true as the workload makes it, and is not kept across a restart.
- **Cancelling** stops the job at once: its running tasks are told to stop, their slots are free again, and nothing more of it is handed out. A cancelled job is over, like one that succeeded or failed, and stays on record.
- **Timeouts.** With `--timeout`, an attempt at a task that runs longer is stopped and counts as a failure, so it is tried again up to the usual three times.
- **A job that fails stops its other tasks** the same way, rather than leaving them to finish for nothing.

## Verifying results

A coordinator takes what a worker returns as it is, unless the job asks otherwise. A job submitted with `--verify N` has each of its tasks run by `N` different workers, and takes a result only once `N` of them have returned the same one.

```sh
bin/sisyphusd job submit --verify 2 --params '{"from":0,"to":100000000}'   # each task on two workers, which must agree
bin/sisyphusd job logs <job-id>                                            # who returned what, and who disagreed
```

- **What "the same" means.** Two results are the same if the outputs are the same bytes and the tasks stored the same blobs, by CID. Stored data is named by its content, so two workers that wrote the same file named the same CID.
- **Different workers.** A task goes to `N` workers at once, each a different node. A worker that has returned a result for a task, or is running it, is never given that task again, so the results that settle a task come from `N` different nodes.
- **When they agree**, which is the usual case, the task has succeeded with that result. The task is shown as the first of those workers'.
- **When they differ**, the task goes to one more worker that has not been asked, for each result the larger side still lacks, until some result has been returned by `N` workers. With `--verify 2`, two that differ are settled by a third. A task takes in at most `2N-1` results. If no result can reach `N` within that, or no connected worker is left that has not been asked, the task fails and the job with it. The error names each worker and a short digest of what it returned.
- **Each disagreement is an event.** `job logs` shows `task-result` for every result with the worker that returned it, `task-disagreed` with the workers on each side when the results in for a task differ, and `task-succeeded` with the workers that agreed and any that returned something else. The job's [record](#a-jobs-record) has the same for good: each receipt lists every result by worker, with a digest and whether it was one of those that settled the task.
- **A failed attempt is not a disagreement.** An attempt that fails, times out (`--timeout`) or is lost with its worker returned no result. It is tried again, by another worker if one could run it and otherwise by that one, and counts towards the usual three failures a task may have.
- **A job that asks for more workers than there are is refused** when it is submitted: `--verify 3` needs three connected workers that could take the job, counting those that are busy. If workers leave afterwards, a task that still needs one waits for it, as a task of any job waits for a worker; the coordinator cannot tell a worker that has gone for good from one that is restarting. Cancel the job if nobody is coming.
- **With no `--tasks`**, a job is split into one task for every `N` free slots, since each task takes a slot on `N` workers. `--mode full-worker` runs the whole job as one task on `N` workers.
- **A restart loses nothing.** The results in are kept in the database. A coordinator that is restarted asks only the workers that had not answered.
- **Logs and progress.** Every copy of a task logs its own lines, each shown with the worker it came from. A task's progress counts the results it has and the furthest of its running copies.
- **A graph hands it on.** A job made of jobs that is verified has each of its steps verified, by as many workers, and with the same share of each step's tasks if it asked for [spot checks](#spot-checks). The graph itself is taken in whoever is connected; a step that cannot be verified when it begins fails the graph and says why.

**What it costs.** `N` times the work, and more when workers differ. A task is as slow as the slowest of the workers it needs.

### Spot checks

Verifying every task multiplies the work. With `--verify-share`, a job has only a share of its tasks verified and the rest run once:

```sh
bin/sisyphusd job submit --verify 2 --verify-share 0.25 --tasks 16 --params '{"from":0,"to":100000000}'   # four tasks on two workers each, twelve on one
```

- **Which tasks is not known beforehand.** That share of the tasks, rounded up to a whole task so that at least one always is, is drawn at random when the job is submitted. A task that is verified and one that is not reach a worker looking the same.
- **A worker that is caught is no longer taken at its word.** When a verified task is settled against what a worker returned, every task of the job that worker ran unverified, or is running, is verified after all: what it returned stands as one result, and the task goes to as many more workers as it now lacks. A task that had succeeded on that worker's word is no longer succeeded, and the job does not finish until it is settled. Whatever of the job that worker is handed afterwards is verified too. `job logs` shows each such task as `task-rechecked`.
- **`job get` and the job's record say which tasks were verified.** `job get` marks each task `verified by N workers` or `not verified`, and each receipt in the record has `verify`: how many workers the task was held to, `1` for one that was run once.
- **What it costs** is about `1 + share × (N - 1)` times the work while nobody is caught: a quarter more for `--verify 2 --verify-share 0.25`.

What a spot check does not do:

- **It does not catch every wrong result.** A worker that returns a wrong result for one task in a hundred is caught only if that task is one of those verified. A share of `s` catches a single wrong result with probability `s`, and a worker that is wrong on `k` of the tasks it ran with probability `1 - (1 - s)^k`, less whatever it gets right. What is not caught is in the job's result.
- **It does not shut a worker out.** A worker that is caught is still given tasks, in this job and the next. What is carried over to the next job is its [standing](#a-workers-standing): a count of its results, and a probation during which a job with spot checks verifies whatever it is handed.
- **A member that is both a worker and a client of the pool can tell.** Whoever may read a job can see which of its tasks went to two workers. Workers that are only workers cannot read jobs.
- **Workers that agree to lie** are no more caught by a spot check than by verifying everything.

### A worker's standing

What verified tasks show of a worker outlasts the job. The coordinator keeps three numbers for each worker, by its node ID:

- **Agreed and outvoted.** When a task that `N` workers had to agree on succeeds, each worker whose result was one of those `N` has one more result that agreed, and each worker that returned something else has one more that was outvoted. A task is counted once, when it is settled.
- **What is not counted.** A task that fails because its workers could not agree counts for nobody: nothing says which of them was right. An attempt that fails, times out or is lost returned no result. A task that was run once and taken as it was shows nothing, unless it is verified after all and settled, and then it counts like any other. A job that is not verified counts for nothing.
- **Probation.** A worker that is outvoted is on probation until ten results it returns after that have agreed. Being outvoted again starts the ten over.
- **What probation does.** In a job with `--verify-share`, a worker on probation is treated as one already caught in that job: a task handed to it is verified, even if it was one of those to be run once, and goes to as many more workers as that takes. `job logs` shows it as `task-rechecked`, saying that the worker was outvoted in another job and how many results it still owes.
- **What probation does not touch.** A job that verifies every task treats the worker as it treats any other. A job without `--verify` still takes its word: it asked for nothing to be checked, and probation does not change that. Only results for verified tasks work a probation off, so in a pool that runs no verified jobs it stays.
- **Where it shows.** `sisyphusd nodes` prints a line under the table for each connected worker that has any, such as `liar: results for verified tasks: 3 agreed, 2 outvoted; on probation, 7 more to agree`. The desktop's API gives the same in `ListWorkers`, and the MCP server in `pool_status`.
- **It is kept in the node's database**, so it survives a restart. It does not go when the jobs it came from are forgotten (`--keep-jobs`), and nothing makes it lapse with time.

What it does not do:

- **It is not a score, and nothing is scheduled by it.** A worker on probation is handed tasks as it was before, in the same order as the others. No worker is ranked, charged or banned, and there is no stake to lose. Removing a worker is still up to the pool's owner (`pool remove`).
- **Being on probation does not mean a worker lied.** A worker outvoted by workers that agreed to lie lands on probation, and so does an honest one whose work does not give the same result every time, when two others happened to agree. What it costs is the pool's time: the tasks such a worker is handed in a job with spot checks are run by `N` workers.
- **It goes by node ID, not by person.** A member that is removed and invited again under a new key starts clean, with no counts and no probation. What was counted under its old ID stays there, and comes back only if that same key is admitted again.
- **It does not go back over other jobs.** A task a worker was handed before it went on probation, in another job running at the same time, is not verified after all. Only the job that caught it does that.

**What it is for, and what it is not.**

- **It is for work that gives the same result every time it is run.** `primes`, `wordcount` and a container that is deterministic do. Work that involves a language model (`chat`, `prompts`, `embed`), `transcode`, and containers that use the time, random numbers or the network may not: two honest workers then differ, and the job fails saying so. Nothing here makes such work comparable.
- **A private job is verified like any other.** Every worker asked is given the job's own key, and sealing gives the same blob for the same key and data, so workers that did the same work stored blobs with the same CIDs. Asking more workers shows the job's data to more of them. In the job's record, what each worker returned is not given by its digest but by a commitment only the job's key can check (`receipts/N/results/M/digest_commitment`), since a digest would tell someone who guessed a task's output that they were right; which workers agreed is still there to read.
- **It does not protect against workers that agree to lie.** `N` workers that return the same wrong result are believed. Nor does it protect, as it is, against one operator running several nodes: different node IDs are not different people. A coordinator started with `--verify-distinct-addresses` makes that harder; see [below](#workers-at-different-addresses).
- **It judges workers only this far.** A worker that was outvoted is named in the events and the record, has that result counted in its [standing](#a-workers-standing), and is on probation until ten of its results have agreed. Nothing else happens to it: no score that decides who is given work, no stake and no ban. Removing it is up to the pool's owner (`pool remove`).
- **It checks what workers return, not how the coordinator combines it.** The coordinator still splits the job and aggregates the outputs itself.


### Workers at different addresses

Two workers on one machine agree with each other because one person runs both. A coordinator started with `--verify-distinct-addresses` hands a verified task's copies only to workers that connected from different network addresses.

```sh
bin/sisyphusd run --role coordinator --verify-distinct-addresses
```

- **What counts as one address.** An IPv4 address as it is; an IPv6 address by its `/64`, which is what one site is usually given. It is the address the worker's connection to the coordinator comes from, so it is the coordinator that says where a worker is, not the worker.
- **Too few is refused at once.** A job with `--verify N` is refused unless workers at `N` different addresses could take it, and the refusal says it is "counting those at one address as one". A task whose workers disagree asks one more, at an address none of the others is at, and fails if there is none.
- **It is for a pool with strangers in it.** Workers behind one router, in one office or on one machine share an address and count as one, so in a pool of your own machines on one network it leaves nothing to verify with. That is why it is off unless asked for. Workers that reach the coordinator through one relay or one VPN exit share an address too.
- **It does not stop someone with several addresses**, and it does not stop two people who agree to lie. It raises what cheating takes from a second process to a second place on the network. [`--audit-share`](#checked-by-the-coordinator-itself) is what catches workers that agree.
- **Jobs that are not verified are not changed**, and nor is who a spot check or an audit picks.

### Checked by the coordinator itself

`--verify` needs at least two workers and is fooled by two that agree to lie. `--audit-share` needs neither a second worker nor anyone's honesty but the coordinator's:

```sh
bin/sisyphusd job submit --audit-share 0.2 --params '{"from":0,"to":100000000}'   # about a task in five is run again by this node
bin/sisyphusd job logs <job-id>                                                   # which, and how each came out
```

- **What it does.** As each task's result comes in, the coordinating node decides by chance, with the probability given, whether to check it. If so the result is not taken yet: the node runs the task itself and compares, as `--verify` compares. The same, and the worker's result is taken and counts for its [standing](#a-workers-standing). Different, and the node's own result is the task's, the worker is counted as outvoted and put on probation, and the job's events say so.
- **No worker knows which tasks.** The choice is made after the result is in. A worker that returns one wrong result in a job audited at a share of 0.2 is caught one time in five, each time.
- **It costs the coordinating machine the work** of every task it checks, on top of what the worker did. A share of 1 has the node do the whole job again.
- **It is for a node that is also a worker**, and for workloads it runs: a node started with `--role coordinator`, or one asked to audit `container` jobs without `--containers`, refuses, since it cannot run the task.
- **It is for work that gives the same result every time**, like `--verify`. A task the node fails to run is not held against the worker: its result is taken, and the events say it went unchecked.
- **Not with `--verify`.** A job asks other workers to check, or the coordinator, not both.
- **A private job is audited like any other**: the node has the job's key.
- **The job's record says it was audited**, with `audit_share` among what was asked. Which tasks were picked and how each came out is in the job's events (`job logs`), which are kept with the job and are not part of the record.
- **A job made of steps** hands its audit share to each step.

## A job's record

When a job is over, whether it succeeded, failed or was cancelled, its coordinator writes its history as linked data: a few small nodes that name each other, and the job's stored data, by content ID. The ID of the root names the whole of it. Change anything in the history and the root's ID changes. That one ID is what a chain would commit to (issue #20).

```sh
bin/sisyphusd job record <job-id>            # the root's ID on a line, then the record's nodes as one JSON object, by ID
bin/sisyphusd job record --verify <job-id>   # also work it out again from the job, and fail if the two differ
bin/sisyphusd job record --key-file job.key <job-id>   # a private job: verify, then check its commitments with the job's key
```

A record is made of these nodes:

| Node | What it holds |
| --- | --- |
| root | `kind` (`sisyphus-job-record`), `version` (1), the job's ID, whether it is `private`, and links to the others. A step of a graph has `parent`: the graph's job ID and the step's name. |
| manifest | What was asked: `workload`, `params`, `mode`, `requirements` (tasks asked for, task timeout, memory, graphics cards, `verify` for a job that was [verified](#verifying-results), and `audit_share` for one the coordinator [checked itself](#checked-by-the-coordinator-itself)), links to the `inputs` it read, the node ID of the `submitter` (absent for a job the node gave itself, from the desktop app or the planner), `created_at`. |
| a receipt for each task | The task's number, how it ended, its `output`, how many attempts failed, and every attempt in order: which worker (node ID and name), how it went, and why it failed if it did. For a verified job also `results`: for each result a worker returned, the attempt, the worker, a `digest` that is the same for results that are the same, and whether it `agreed` with the result the task was settled by. |
| result | How the job ended, its `result` or `error`, links to the `outputs` it stored and to the `intermediate` data that passed between its tasks, `finished_at`. |
| graph | Only for a job made of jobs: each step's name, its job's ID, and a link to that job's record. |

- **It is ordinary IPLD.** Each node is DAG-CBOR and its ID is a version 1 CID of it with SHA-256. On a node started with `--kubo`, `ipfs dag get <root>/manifest/workload` reads it, following the links by name. The JSON printed is DAG-JSON: a link is `{"/": "<cid>"}` and bytes are `{"/": {"bytes": "<base64>"}}`.
- **The same job always gives the same record.** It is worked out from the job and nothing else, so the ID is known the moment the job ends: `job get` prints it, and the job carries it as `record_cid`. Storing the nodes follows at once; if that fails, or the store has lost them, they are stored again when the record is asked for. A restart changes nothing.
- **`--verify` checks the record against the job** as the coordinator holds it now, and each node read is checked against its ID. It finds a job whose row in the database, or whose record in the store, was changed after the job ended. It proves nothing to someone who does not trust this coordinator: a record is not signed. Signing the root with the node's key is future work.
- **Parameters, a result or a task's output longer than 16 KiB** are not held in the node but linked, by the CID they have as a stored blob, and stored beside the record for the retention period.
- **A record outlives the job's data.** It is kept for as long as the job is on record (`--keep-jobs`), while the data it links is kept for `--retain`. After that the links name data the node no longer holds. A graph's steps stay on record for as long as the graph does.
- **Times that are not kept across a restart are left out**: when each attempt began and ended. So is which task read or stored which blob; the record has that for the job as a whole.
- **A job from before records were kept has none.** `record_cid` is empty for it.

**What a record reveals.** A record sits in the coordinator's store, and on a node with `--kubo` in the pool's private network, where any member that knows or guesses its ID can fetch it. An ID cannot be guessed, and is told to clients of the coordinator with the job. Whoever has it reads everything in the table above: the workload, the parameters, the result, each task's output, the error messages, the node IDs and names of the submitter and of every worker that was handed a task, the times, and the IDs of the job's data.

For a **private job** the record says so and leaves out everything that was not sealed: the parameters, the result, each task's output, and every error message, since a message may quote data. It keeps the workload's name, the requirements, who ran what and how it went, the times, and the IDs of the sealed inputs and outputs, which are of no use without the key.

In place of each value left out, the record of a private job holds a **commitment** to it: 32 bytes that fix the value without showing it. So the record still commits to the job's result, and whoever holds the key can later show that a given result is the one the job produced.

| Where | What it commits to |
| --- | --- |
| manifest, `params_commitment` | The parameters. |
| each receipt, `output_commitment` | The task's output. |
| each attempt in a receipt, `error_commitment` | Why the attempt failed. |
| each result in a receipt of a verified job, `digest_commitment` | The digest of what the worker returned. Results that were the same do not have the same commitment; `agreed` says which side each was on. |
| result, `result_commitment` and `error_commitment` | The job's result, and its error. |

- **What a commitment is.** HMAC-SHA-256 of the job's ID, the commitment's path in the record (such as `receipts/0/output_commitment`) and the value. The ID and the path are each preceded by their length, as eight bytes, big-endian. The HMAC key is 32 bytes derived from the job's sealing key by HKDF-SHA-256 with no salt and the label `sisyphus job record: commitments`, so it is not the key the data is sealed with.
- **Without the key it reveals nothing about the value**: not the value, and not whether a guess at it is right. A plain hash would let anyone confirm a guess at a small result such as `{"count":25}`. It does not hide that there is a value to commit to: every private record has every commitment in the table, and an empty value, such as the error of an attempt that succeeded, is committed to like any other.
- **Equal values do not give equal commitments.** The job's ID and the path go into each, so the same result in two jobs under one key, or the same output from two tasks of one job, gives commitments that cannot be told to be of equal values.
- **They are made when the job ends**, while the coordinator still has the key, and saved with the job. The key is not saved with a finished job. The record is worked out again from the saved commitments, so `--verify` works on a private job without the key, before and after a restart.
- **`--verify` without the key does not check the values.** The record holds the commitments and not the values, so a result changed in the coordinator's database after the job ended leaves the record as it was. Only the key finds that.
- **`--key-file` checks them.** It takes the job's key, the file `job submit --key-file` was given, or the node's `private.key` for a job made private in the desktop app. It verifies the record, then sends the key to the coordinator, which works each commitment out again from the parameters, result, outputs and errors it holds and says of each whether it `matches` or `does not match`. The coordinator uses the key for that and does not keep it. The command fails if any does not match. That happens when the key is not the job's, in which case none match, or when a value has changed since the job ended. It says nothing against the record itself, which was verified first.
- **A record with no commitments has nothing to check**, and `--key-file` says so: the record of a job that is not private, or of a private job that ended before commitments were made, whose record is as it was.

## Restarting a coordinator

A coordinator keeps its jobs in `node.db`, a SQLite database in its data directory, so stopping it loses nothing:

- **Finished jobs** are still there to be asked after, with their results.
- **Unfinished jobs** are taken up where they were. Tasks that had finished stay finished. Tasks that were running go back to wait for a worker, and that is not counted against them, however often it happens. A task of a [verified](#verifying-results) job keeps the results it had, and only the workers that had not answered are asked again. If every task had finished and only combining their outputs was cut short, the combining is done again, with no worker needed.
- **A job's data** stays pinned across the restart for as long as the job takes.
- **Workers** reconnect by themselves once the coordinator is back.
- **Storage followers** go on holding what they held. A coordinator that comes back without its stored data takes it back from them; see [Copies on other nodes](#copies-on-other-nodes).

A job's arrival and its finish are written through to the disk before anyone is told of them. The steps in between are not, so a power cut can lose the last few; the tasks concerned run again. A coordinator started without a workload that an unfinished job needs fails that job, and says why.

The database holds the rest of what a coordinator must not forget as well: which nodes have been admitted, and the invitations issued and not yet used, so an invitation is still good after a restart. Invitations are recorded by a hash of their token, so reading the database admits nobody. A coordinator from before this was so finds its old `access.json`, takes its members in, and sets the file aside as `access.json.imported`.

A finished job can be asked after for 30 days, or whatever `--keep-jobs` says; `--keep-jobs 0` keeps them for good. That is about the record of the job. How long its data is kept is `--retain`, described under [How long data is kept](#how-long-data-is-kept).

A node's own record of the coordinators it has joined, `known.json`, stays a file. Every command reads it, on machines that run no coordinator and have no database, and it is small enough to mend by hand when a coordinator's address changes.

The database also records every attempt at every task: which node, and how it went. A finished job's [record](#a-jobs-record) is written from it. It goes when the job does.

## The desktop app

[`apps/sisyphus`](../sisyphus/README.md) is a desktop client that shows a node and its peers. It talks to a node on the same machine through the local API, `sisyphus.node.v1` in [`proto/sisyphus/node/v1`](../../proto/sisyphus/node/v1/node.proto). The local API is off unless asked for:

```sh
./sisyphusd run --name alpha --api-listen 127.0.0.1:50051
```

`127.0.0.1:50051` is where the desktop app looks. The address must be a loopback one: the local API has no TLS, and is for programs on this machine only.

What it shows, in this daemon's terms:

| The desktop's word | What it is here |
| --- | --- |
| peer | A node of this node's pool, or one its libp2p host has found: on the same network, through nodes it knows, or because it was given the address. |
| connected | The node has a live connection to this one right now, as a worker or between the two hosts. |
| gives work, takes work | The two sides of this node's trust in the peer: that it gives the peer tasks, and that it takes the peer's. See [Trust](#trust) below. |
| trusted for compute | Either of the two. The call that sets it sets both, for clients that know only the one switch. |
| working for you, you work for it | Which way work is flowing now: the peer is connected as a worker of this node, and this node is working for the peer. Both need trust on both sides. |
| connect to an address | Connects this node's host to the node at an address ending in `/p2p/<node ID>`, which is the form a node lists its own in, and notes it in the address book. |
| bootstrap peers | The node's address book: the addresses it connects to each time it starts. |
| files | What was put in the pool's store through the app: kept until removed, listed by name, and given to jobs by CID. The same store `blob put` and `blob get` use. |
| invitation, members | `pool invite`, `pool members` and `pool remove`, from the app. |
| private | A file or job sealed with the key the node keeps in `private.key` in its data directory, made the first time one is asked for. The same as `--key-file` with that file. Lose the file and what it sealed is lost. It is one key for everything the node makes private: a worker handed any private job from this node can open all of its private files and the data of its other private jobs. |
| join a pool | `pool join` and a restart with `--join`, without the restart: the node redeems an invitation, says it takes that node's work, and starts on it. |
| country | Where the node's address is registered, and each peer's, from a table the daemon carries (`packages/geo`, made from the regional Internet registries' published allocations). Nothing is asked of anyone. It is empty for a node that knows itself only by a home-network address, until other nodes have told it the address they see it at; `--locate-country` then asks ipapi.co instead, which thereby learns the address, so that is not done unasked. The country is the one of registration, which now and then is not where the machine is. `scripts/update-geo.sh` remakes the table. |

With `--web-listen`, the same API is served to browser clients as Connect and gRPC-Web. Add `--web-ui-dir <renderer-build-directory>` to have this node serve the React UI from the same origin too. It is loopback-only by default. LAN access requires `--web-lan` and a private IP assigned to this machine (never `0.0.0.0`); it uses plain HTTP, so pairing codes and sessions are not encrypted on the network. A local browser can create one-time pairing codes/QRs and revoke paired devices in Settings. [`docs/desktop-backend.md`](../../docs/desktop-backend.md#from-a-web-page) has the details.

Anything on the machine can read what the node is, who it knows and who works for it. Changing something, and reading jobs, conversations or what files contain, needs the token in `api.token` in the data directory, which the daemon makes on first use and only its own user can read. The desktop app looks for it in the daemon's default data directory, or in the file named by `SISYPHUS_API_TOKEN_FILE`.

## Finding other nodes

A node looks for others and lets them find it, unless started with `--discovery off`:

- **On its own network**, by multicast DNS. Two nodes on one LAN find each other within a second or two with nothing configured.
- **Through the nodes it knows**, by a distributed hash table (Kademlia) that every node both uses and serves. A node that knows one other can find the rest, and can reach a node knowing only its ID.
- **From its address book**, the addresses it connects to each time it starts. `--bootstrap <address>,<address>` adds some for one run; the desktop's "connect to peer" and bootstrap list add them for good.

The names it uses for both are Sisyphus's own, so nodes find each other and not every other libp2p program in reach.

**Being found is not being let in.** With discovery on, any libp2p node may connect to a node's host. It can then see the node's ID and addresses and take part in the hash table, and that is all. Everything else checks who is asking: jobs, stored data, the relay and the private IPFS network are for the pool's members. With `--discovery off` a node's host accepts connections from its pool's members only, and finds nobody.

## Trust

Trust in another node for compute has two sides, set separately:

- **Giving it work.** The node may work for this one: it is a worker in this node's pool.
- **Taking its work.** This node will work for it.

Work flows from one node to another when the first gives and the second takes. Nobody is put to work by someone else's choice, and nobody is given tasks by a node they did not agree to work for.

| To do this | From the desktop | From the command line |
| --- | --- | --- |
| Give a node work | "Give it work" beside the peer | `pool invite` here and `pool join` there, for a node not yet known; nothing for one already found, use the desktop |
| Stop giving it work | the same button | `pool remove <node ID>` |
| Take a node's work | "Take its work" beside the peer | `pool work-for <node ID>` |
| Stop taking it | the same button | `pool work-for --stop <node ID>` |

`pool members` lists both: the nodes admitted, and the nodes this one will work for.

**How two nodes come to work together**

- **Two nodes that each run a pool, on one network.** Each finds the other in its peer list. Each owner chooses what they want of the other: give, take, or both. Wherever one gives and the other takes, work starts within a few seconds, with neither node restarted. Both choosing both means each works for the other.
- **An invitation.** `pool invite` on a coordinator and `--join` on a worker-only node, as always. The coordinator gives; the worker, started for that coordinator, takes.
- **A worker-only node and a second coordinator.** A node started as one coordinator's worker can take another's work as well: that coordinator gives it work, and it is told to take it, from its desktop or, on a node that also runs a pool, `pool work-for`.

**What holds whichever way it was set up**

- **Either side ends it.** A node that stops giving cuts the other's worker off; a node that stops taking stops working. Both are at once.
- **It takes effect at once.** When a node's giving changes, it sends word to the node concerned over libp2p, which looks again on hearing. Should word not arrive, each asks again once a minute anyway.
- **A node's slots are shared, fairly.** However many nodes it works for, it runs no more tasks at once than `--slots`. A coordinator may hand it more than that, and those wait their turn. When a slot comes free it goes to the pool with the fewest of its tasks running on this node, so one pool's long queue does not hold up another's single task.
- **Each pool's data is kept apart.** What a node fetches while working for another is kept under `guest/<that node's ID>` in its data directory, away from its own pool's store and from every other node's.
- **With `--kubo`, it joins the other pool's private IPFS network too.** A private network has a key of its own, so the node runs one more Kubo daemon for each pool it works for, on a repository under that same directory, and follows that pool's key when it changes. Each costs about as much memory as the node's own. If the other node runs no such network, the work is done without one.
- **The desktop shows which way work is flowing**: beside each peer, "Working for you" and "You work for it", whichever hold.
- **`--work-for-trusted=false`** has a node take nobody's work but that of the coordinator it was started for, whatever its list says.

What this does not do yet:

- **Word of a change can be sent by any node that can connect.** All it does is make this node ask sooner, so the worst a stranger can do with it is cause some needless asking.
- **Nothing limits what a node gives.** A node that gives another work gives it any task; there is no "only these workloads" or "only this much".
- A node works for at most sixteen others at once.

## Coming from the Rust daemon

Before this daemon there was one written in Rust, which kept a node's key, address book and trusted nodes in one database, `node.sqlite3`, in the same data directory. A node that ran it carries on here as itself:

- **The same key, so the same ID.** The first command that needs the node's key takes it from that database rather than making a new one.
- **The same address book and the same trusted nodes**, carried over the first time the node runs a pool. That daemon's trust had one side; a node it trusted is here both given work and taken from.
- The old database is then set aside as `node.sqlite3.imported`. Nothing is deleted.

If that database is there and cannot be read, the node stops and says so. It does not quietly start as a different node.

This was checked against a database made by the Rust daemon itself, which is kept with the tests.

## Examples

[`examples/`](../../examples/README.md) has a script that runs a pool on this machine and leaves it up, scripts that run each workload against a pool, and a sample input. [`docs/multi-machine-test.md`](../../docs/multi-machine-test.md) is a guided test of a pool across real machines.

## Workloads

- **`container`** runs a container image, once for each task of the job. This is how real programs run on a pool: anything that can be put in an image. See [Running containers](#running-containers) below.
- **`transcode`** re-encodes a video. It is the first workload that does a job people have: see [Transcoding video](#transcoding-video) below.
- **`primes`** counts the primes in `[from, to)`. Parameters and results are a few bytes and travel inline. It is a stand-in that exercises the pool.
- **`wordcount`** counts word frequencies in a stored file. It exercises the storage path: the input is fetched by CID, each task reads only its byte range, task outputs are stored as blobs, and the result is a blob too. The result's CID is the same however the job is split, so two runs can be compared by CID alone.

### Running containers

```sh
bin/sisyphusd run --role worker --coordinator <addr> --containers ...     # a worker that will run them
bin/sisyphusd job submit --workload container --tasks 8 \
    --params '{"image":"alpine:3.20","command":["sh","-c","echo I am task $SISYPHUS_TASK_INDEX"]}'
examples/container.sh                                                    # the same, and prints each task's output
examples/render.sh                                                       # a picture rendered in strips across the pool
```

- **A worker runs containers only if its owner says so**, with `--containers`, and needs Docker. **This is real authority over the machine:** whoever may submit jobs to the pool can run any image with any command there. Give it only on pools whose clients you would let log in.
- **One copy for each task.** Every task runs the same image and command and is told which copy it is, in `SISYPHUS_TASK_INDEX` (from 0) and `SISYPHUS_TASK_COUNT`. A program that does a share of some larger work takes its share from those.
- **Input.** `"input": "<CID>"` gives every task that stored file at `/input/data`, read-only.
- **Output.** What the command prints is stored and its CID returned for each task, and so is every file it leaves in `/output`, up to 4 GiB and 1,024 files for a task: a task that leaves more fails. `blob get <CID>` fetches them. What it prints is also the task's log, live, in `job logs`, of which a job keeps 20,000 lines; all of it, up to 16 MiB for a task, is in what was stored.
- **Limits.** `"memory_mb"` and `"cpus"` cap what each task may use, and unlike a node's offer these are enforced, by Docker. A task has no network unless the job says `"network": true`. It runs as the user the node runs as: not as root unless the node itself does, and with none of the capabilities Docker usually gives a container. Where the system has no user numbers, as on Windows, it runs as the image has it, with Docker's usual capabilities. Either way it cannot gain privilege it did not start with (`no-new-privileges`), so a setuid program in the image does nothing more than any other, and it may have 4,096 processes at once. The node's user has no home in the image, so the command's `HOME` is `/tmp`, which it can write to, unless the job's `env` gives another: `pip install` and the like, which install for the user, then work.
- **Holding tasks in further.** A container shares this machine's kernel, which is the weak point when the work is a stranger's. Two flags on the worker ask more of Docker:
  - `--container-runtime runsc` runs every task with the runtime named in place of Docker's usual: [gVisor](https://gvisor.dev)'s `runsc`, which stands between the task and the kernel, or Kata Containers' runtime, which gives each task a small virtual machine. The runtime is installed and made known to Docker by you; the node checks Docker has it and does not start otherwise, so it never runs tasks less held in than you asked. Neither has been tried here: this machine has only Docker's own runtime, and a graphics card complicates both.
  - `--containers-read-only` gives a task nowhere to write but `/output` and `/tmp`, with `/tmp` held in memory and counted against the task's memory. Nothing in the image can be changed. An image that writes elsewhere fails; `transcode` and what the examples run do not.
- **Stopping.** Cancelling the job, or a task running past the job's `--timeout`, kills the container.
- **A job is taken in by any coordinator**, and waits until a worker that runs containers is connected.
- **Tasks of a job that run on one machine share one copy of its input**, fetched once. Tasks of different jobs share nothing.

- **Graphics cards.** `"gpus": 1` gives each task that many of the machine's cards, which needs Docker set up for them (the NVIDIA Container Toolkit). Submit the job with `--min-gpus` as well, so that its tasks go only to machines that have them. This has not been tried on a machine with a card.

What it does not do yet: images are pulled by Docker when first used, from wherever the image name says, so a worker needs to reach that registry and trust it. To be sure of what runs, name the image by digest (`alpine@sha256:...`), which Docker checks.

### Transcoding video

```sh
cid=$(bin/sisyphusd blob put holiday.mov)
bin/sisyphusd job submit --workload transcode --params "{\"input\":\"$cid\",\"height\":720}"
examples/transcode.sh holiday.mov 720       # the same, and fetches the result
```

The video is cut into as many stretches as the job has tasks. Each task encodes one with ffmpeg, in a container, and the stretches are joined into one video: H.264 with AAC sound, in an MPEG transport stream (`.ts`), which players open and `ffmpeg -i out.ts -c copy out.mp4` puts in an MP4 without encoding again.

| Parameter | Meaning |
| --- | --- |
| `input` | The CID of the stored video. Anything ffmpeg reads. |
| `height` | Scale the picture to this many pixels high, keeping its shape. An even number. Left out, the size is kept. |
| `crf` | Quality, from 1 (best, largest) to 51. Default 23. |
| `preset` | Time against size, from `ultrafast` to `veryslow`. Default `medium`. |
| `segments` | How many stretches. Default: one for each free slot. |

- **It runs on workers started with `--containers`**, like any container job, but the image and the command are fixed: this workload cannot be made to run anything else.
- **Nothing to install but Docker.** A worker builds the image the first time it is needed, from two lines (Alpine and its ffmpeg package). That takes about a minute and the network, once.
- **When it helps.** ffmpeg already uses every core of one machine, so cutting a video up gains little there: on one 32-core machine, a one-minute 1080p clip at the `slow` preset took 43 s as one task and 31 s as eight, and a light encode was no faster cut up. The gain is across machines, where each encodes its stretch at the same time. Every worker fetches the whole video to encode its part, so a pool on a slow link is better served by fewer, longer stretches.
- **Where stretches meet**, the sound can overlap by a few hundredths of a second, and players step over it. Every frame of the picture is there, once.
- **Private jobs**: the workload reads and stores through the same sealed store the others do, so a sealed video and a job with its key should work; this has not been tried.

## Layout

| Path | What it holds |
| --- | --- |
| `proto/sisyphus/v1` | Message and service definitions. Edit these, then run `make proto`. |
| `proto/sisyphus/node/v1` | The local API the desktop app uses. |
| `packages/protocol` | Go code generated from `proto/`. Do not edit by hand. |
| `packages/job-model` | Job and task state machines. No I/O. |
| `packages/jobrecord` | A finished job as linked data: building the record and reading it back. |
| `packages/runtime` | The `Workload` interface and built-in workloads. |
| `packages/storage` | Content-addressed blob store with IPFS-compatible CIDs. |
| `packages/kubo` | Running a Kubo daemon beside the node, and calling it. |
| `packages/ipfscluster` | Running an IPFS Cluster peer beside the node's Kubo, and calling it. |
| `apps/sisyphusd/coordinator` | Scheduling, retries, aggregation, worker connections. |
| `apps/sisyphusd/worker` | Connects to a coordinator and executes tasks. |
| `apps/sisyphusd/api` | gRPC server wiring and the client-facing services. |
| `apps/sisyphusd/access` | Who has been admitted and in what role; invitations; checking every call. |
| `apps/sisyphusd/tunnel` | Carrying a TCP connection inside a gRPC stream. |
| `apps/sisyphusd/replication` | Copies of a coordinator's pinned data on storage followers: who holds what, and fetching it back. |
| `apps/sisyphusd/p2p` | The node's libp2p host: one port shared with gRPC, the relay, discovery, reaching a node by its ID. |
| `apps/sisyphusd/planner` | The loop in which a model reasons, has the pool compute, and reads the result. |
| `apps/sisyphusd/inference` | The pool's language models offered as an OpenAI-style service. |
| `apps/sisyphusd/mcpserver` | The node offered to AI agents as a Model Context Protocol server. |
| `apps/sisyphusd/pinning` | The IPFS Pinning Service API: the node as a service, and as a client of one. |
| `packages/ai` | Talking to language models: Ollama, OpenAI-style services and Anthropic. |
| `packages/hardware` | Finding out what a machine has: processor, memory, graphics cards. |
| `packages/identity` | Node keys, IDs, and the TLS settings built from them. |
| `packages/names` | The signed records behind names: making, checking and comparing them. |
| `packages/nodedb` | The node's SQLite database: jobs, tasks, attempts and what they returned, members and invitations. |
| `packages/storage` | Content-addressed blob store, pins, garbage collection. |

`make proto` needs `protoc` on your path and the plugins from `make tools`.

## Known limits

- A task may fail three times, and losing its worker counts as a failure. Losing its coordinator does not. Its next attempt goes to a worker it has not gone wrong on, waiting for one that is busy if it must, and back to one it has only when there is no other that could run it.
- A coordinator accepts whatever result a worker returns, unless the job asks to be [verified](#verifying-results), and then it believes any `N` workers that agree. Verification is for deterministic work only. Admit only workers you trust.
- A node is remembered by address. If a coordinator's address changes, its workers and clients must join again.
- A node's key cannot be changed without becoming a different node, and there is no way to stop a copied key being used other than removing that node.
- Encryption hides what nodes say to each other, not that they are talking, how much, or when.
- If a node using `--kubo` is killed outright rather than stopped, its Kubo daemon keeps running and must be stopped by hand before the node will start again. With `--cluster` the same goes for its cluster peer.
- With `--cluster`, only the coordinator's peer can change what is pinned or find a new holder for it. While the coordinator is down the copies stay as they are.
- A removed node keeps whatever it had already fetched from the pool's private network. Changing the key stops it fetching anything more.
- Run `ipfs` commands against a node's repository only while the node is up. With its Kubo down, including for the few seconds of a key change, the `ipfs` command takes the repository's lock and the node cannot start Kubo until the command ends.
- Whatever is on the pool's private network can be fetched by every member of it. The network keeps outsiders out; it is private jobs, which seal their data, that keep members from reading each other's.
- A job's record is not signed, and its root links every receipt, so a job of ten thousand tasks has a root of about 400 KiB. It is not served by the gateway, which serves files only.
- A node has one name, its ID, and a record published by hand is not renewed: it stops resolving when its lifetime is over.
- The records of names reach the pool's private IPFS network only when they are published; a record that has gone from there is not put back until its node publishes again.
- Without `--max-store-bytes` there is no limit on what a worker or client can upload.
- Without `--replicas` and at least one follower, a pool's stored data is on its coordinator's disk and nowhere else. With them, the coordinator's database and key are still on that disk alone, and without the key the data comes back only by hand and without its expiries.
- The pinning service has one key and speaks plain HTTP. Whoever holds the key can fill the store up to `--max-store-bytes`, and a single fetch can go past it.
- A pool's data can be kept by an outside pinning service only if that service is on the pool's private IPFS network, or has the data from somewhere else.
- A worker downloads a whole input even when its tasks need only part of it.
- Two workers that cannot reach each other exchange data through the coordinator, which is no faster than fetching from it.
- Without `--max-cache-bytes` a worker's cache grows until the disk is full. The limit is not strict: blobs that tasks have open are kept even if they alone exceed it.