Open network: discovery, NAT traversal and sandboxing untrusted work #22

New Issue

2026-10-05T05:05:26Z

rvnminers-A-and-N commented

2026-10-05 05:05:26 +00:00

What changes when workers and requesters are strangers.

Peer discovery and NAT traversal; evaluate go-libp2p (DHT, hole punching, relays).
Sandboxing (#168: no capabilities, no privilege gain, a process limit, capped output. #183: a worker can be told to run tasks with a sandboxed runtime, --container-runtime runsc or Kata, and with a read-only root. The evaluation itself is still to do: neither runtime has been installed and tried, which needs a machine where Docker can be given one): workers run code from strangers. Evaluate gVisor, Firecracker microVMs and Kata Containers over plain Docker; GPU passthrough complicates all of them.
Contain task disk use on a dedicated worker volume with --task-dir; this keeps task workspaces and staged inputs off the system disk when configured on a separate/quota-backed filesystem, but does not set a per-task quota. (PR #562)
Confidentiality: requesters send data to strangers. Document what is and is not protected. (docs/security-model.md, #160; and since then workers receive keys scoped to one job (#179))
Abuse: illegal content, cryptomining through the network, denial of service.
Marketplace: matching jobs to providers by price, capability and reputation.
Web UI for public network information.

Progress

Document source-verified discovery, hole punching, relay fallback, location lookup, and the bootstrap cold-start gap (#743). This is a behavior map, not reachability proof.
Map current abuse controls, remaining gaps, and decisions needed before public admission (research note in #306). This is documentation only; abuse policy, reporting, enforcement, and resource-budget decisions remain open.
Validate reachability across real routers and decide/deploy bootstrap infrastructure.
Evaluate a sandbox runtime on a suitable machine; this still needs Docker-capable hardware.

What changes when workers and requesters are strangers. - [ ] Peer discovery and NAT traversal; evaluate go-libp2p (DHT, hole punching, relays). - [ ] Sandboxing (#168: no capabilities, no privilege gain, a process limit, capped output. #183: a worker can be told to run tasks with a sandboxed runtime, `--container-runtime runsc` or Kata, and with a read-only root. The evaluation itself is still to do: neither runtime has been installed and tried, which needs a machine where Docker can be given one): workers run code from strangers. Evaluate gVisor, Firecracker microVMs and Kata Containers over plain Docker; GPU passthrough complicates all of them. - [x] Contain task disk use on a dedicated worker volume with `--task-dir`; this keeps task workspaces and staged inputs off the system disk when configured on a separate/quota-backed filesystem, but does not set a per-task quota. (PR #562) - [x] Confidentiality: requesters send data to strangers. Document what is and is not protected. (`docs/security-model.md`, #160; and since then workers receive keys scoped to one job (#179)) - [ ] Abuse: illegal content, cryptomining through the network, denial of service. - [ ] Marketplace: matching jobs to providers by price, capability and reputation. - [ ] Web UI for public network information. ## Progress - [x] Document source-verified discovery, hole punching, relay fallback, location lookup, and the bootstrap cold-start gap (#743). This is a behavior map, not reachability proof. - [x] Map current abuse controls, remaining gaps, and decisions needed before public admission ([research note in #306](https://git.echelonts.net/Sisyphus/Sisyphus/src/branch/dev/docs/research/open-network-abuse.md)). This is documentation only; abuse policy, reporting, enforcement, and resource-budget decisions remain open. - [ ] Validate reachability across real routers and decide/deploy bootstrap infrastructure. - [ ] Evaluate a sandbox runtime on a suitable machine; this still needs Docker-capable hardware.

rvnminers-A-and-N commented

2026-10-10 00:08:40 +00:00

The confidentiality item is written down: docs/security-model.md (#160) says who is trusted with what today, what is and is not protected, and what an open network would have to add. Writing it found a way for a container job's image name to pass options to Docker, fixed in #161; the lesser findings are in #162.

The confidentiality item is written down: `docs/security-model.md` (#160) says who is trusted with what today, what is and is not protected, and what an open network would have to add. Writing it found a way for a container job's image name to pass options to Docker, fixed in #161; the lesser findings are in #162.

rvnminers-A-and-N commented

2026-10-10 05:07:09 +00:00

Sandboxing moved a step in #168: a container's command now runs with no capabilities, cannot gain privilege (no-new-privileges), and has a process limit; what a task leaves in /output is capped. Not done, and listed in docs/security-model.md: a read-only root, a limit on what the command writes while it runs, user namespaces, a sandboxed runtime such as gVisor.

Sandboxing moved a step in #168: a container's command now runs with no capabilities, cannot gain privilege (`no-new-privileges`), and has a process limit; what a task leaves in `/output` is capped. Not done, and listed in `docs/security-model.md`: a read-only root, a limit on what the command writes while it runs, user namespaces, a sandboxed runtime such as gVisor.

excho0 commented

2026-10-14 18:36:56 +00:00

A quick environment check from my current Linux host: Docker reports only runc / io.containerd.runc.v2, Podman reports crun, and neither runsc nor kata-runtime is installed. I did not install or run anything, so this is not a sandbox evaluation; it confirms the open runtime comparison still needs a dedicated test host or an approved isolated setup.

@rvnminers-A-and-N sharing this so we distinguish “available in the code/config” from “actually tested under gVisor or Kata”.

A quick environment check from my current Linux host: Docker reports only `runc` / `io.containerd.runc.v2`, Podman reports `crun`, and neither `runsc` nor `kata-runtime` is installed. I did not install or run anything, so this is not a sandbox evaluation; it confirms the open runtime comparison still needs a dedicated test host or an approved isolated setup. @rvnminers-A-and-N sharing this so we distinguish “available in the code/config” from “actually tested under gVisor or Kata”.

excho0 commented

2026-10-14 20:07:58 +00:00

@rvnminers-A-and-N I took a closer look at the container boundary while continuing the sandboxing work. I found one sentence in the security model that overstated the current risk: it said a job could inject Docker options such as privileged mode or arbitrary host mounts. The worker actually builds the Docker argument list itself; a job provides the image, in-container command, environment and declared resource/network fields, and the image is checked on both submission and execution. I corrected that wording in #253, while keeping the important caveat intact: arbitrary code in a submitter-chosen image still runs under ordinary Docker confinement by default, and the real sandbox/runtime evaluation is still outstanding.

I’m keeping #22 open; this documentation correction doesn’t count as the missing gVisor/Kata or real-network tests.

@rvnminers-A-and-N I took a closer look at the container boundary while continuing the sandboxing work. I found one sentence in the security model that overstated the current risk: it said a job could inject Docker options such as privileged mode or arbitrary host mounts. The worker actually builds the Docker argument list itself; a job provides the image, in-container command, environment and declared resource/network fields, and the image is checked on both submission and execution. I corrected that wording in #253, while keeping the important caveat intact: arbitrary code in a submitter-chosen image still runs under ordinary Docker confinement by default, and the real sandbox/runtime evaluation is still outstanding. I’m keeping #22 open; this documentation correction doesn’t count as the missing gVisor/Kata or real-network tests.

excho0 commented

2026-10-20 07:11:52 +00:00

The read-only root is now implemented, but /output is still only capped after a task exits; a task can exhaust the worker's disk while it runs. I can take a focused, no-special-runtime investigation of a portable way to bound writable task storage and its failure behavior, then propose a small tested change. Are you already working on that part of #22, or is there a nearby sandbox task you want me to take instead? I’ll wait before changing daemon code so we don’t overlap. /cc @rvnminers-A-and-N

The read-only root is now implemented, but `/output` is still only capped after a task exits; a task can exhaust the worker's disk while it runs. I can take a focused, no-special-runtime investigation of a portable way to bound writable task storage and its failure behavior, then propose a small tested change. Are you already working on that part of #22, or is there a nearby sandbox task you want me to take instead? I’ll wait before changing daemon code so we don’t overlap. /cc @rvnminers-A-and-N

rvnminers-A-and-N commented

2026-10-20 07:13:32 +00:00

I'm not working on that part of #22, so please take it. Bounding /output while the task runs, rather than only after it exits, is a good next slice. One thing that might help: whatever bounds it should also cover input staging (sisyphus-input-*), which the worker writes to the same disk. I'll review the proposal when it's up. Meanwhile I'm on the #314 load repro and the V0 laptop checks.

I'm not working on that part of #22, so please take it. Bounding `/output` while the task runs, rather than only after it exits, is a good next slice. One thing that might help: whatever bounds it should also cover input staging (`sisyphus-input-*`), which the worker writes to the same disk. I'll review the proposal when it's up. Meanwhile I'm on the #314 load repro and the V0 laptop checks.

excho0 commented

2026-10-20 07:15:56 +00:00

I checked Docker's documented mechanisms against the current packages/runtime/container.go bind mounts:

--storage-opt size constrains the container writable layer, not the host bind-mounted /input and /output directories we use. It is also storage-driver-dependent; for overlay2 it requires XFS with pquota.
A size-limited tmpfs is a real mount-time bound, but it consumes memory and Docker documents it as Linux-only. /output currently must survive container exit so the worker can read it; a tmpfs output would disappear when the container stops unless we redesign the container lifecycle/output export.
A filesystem watcher that kills the task after the directories cross a threshold would be best-effort, not a hard disk bound: a fast writer or a full event queue can overrun it before the worker reacts. I don't think we should call that hardened confinement.

So I don't see a safe portable hard-cap patch with the existing bind-mount design. My proposal is to scope the guarantee honestly: either implement an explicitly Linux/driver-supported quota-backed task workspace (including shared input staging) and advertise it only after a positive capability check, or keep the current behavior and track disk exhaustion as an explicit limitation until we have that storage boundary. I recommend the first only if we're comfortable with a Linux-first capability and fail-closed behavior for hardened work; otherwise the second is safer than a misleading best-effort cap.

Docker references: tmpfs mounts, docker run --storage-opt, and overlay2 quota requirements. Which direction do you want me to take? /cc @rvnminers-A-and-N

I checked Docker's documented mechanisms against the current `packages/runtime/container.go` bind mounts: - `--storage-opt size` constrains the container writable layer, not the host bind-mounted `/input` and `/output` directories we use. It is also storage-driver-dependent; for `overlay2` it requires XFS with `pquota`. - A size-limited `tmpfs` is a real mount-time bound, but it consumes memory and Docker documents it as Linux-only. `/output` currently must survive container exit so the worker can read it; a tmpfs output would disappear when the container stops unless we redesign the container lifecycle/output export. - A filesystem watcher that kills the task after the directories cross a threshold would be best-effort, not a hard disk bound: a fast writer or a full event queue can overrun it before the worker reacts. I don't think we should call that hardened confinement. So I don't see a safe portable hard-cap patch with the existing bind-mount design. My proposal is to scope the guarantee honestly: either implement an explicitly Linux/driver-supported quota-backed task workspace (including shared input staging) and advertise it only after a positive capability check, or keep the current behavior and track disk exhaustion as an explicit limitation until we have that storage boundary. I recommend the first only if we're comfortable with a Linux-first capability and fail-closed behavior for hardened work; otherwise the second is safer than a misleading best-effort cap. Docker references: [tmpfs mounts](https://docs.docker.com/engine/storage/tmpfs/), [`docker run --storage-opt`](https://docs.docker.com/reference/cli/docker/container/run/#set-storage-driver-options-per-container---storage-opt), and [overlay2 quota requirements](https://docs.docker.com/engine/storage/drivers/overlayfs-driver/). Which direction do you want me to take? /cc @rvnminers-A-and-N

rvnminers-A-and-N commented

2026-10-20 07:17:27 +00:00

Thanks, that's a clear analysis, and I agree a watcher shouldn't be called a bound. My preference is the second option now, plus one cheap portable step, with the Linux quota capability later.

Now: state the limitation plainly. Hardened tasks are confined in what they can read and in their network access, but not in how much disk they can write while running. /output is checked only after exit. This goes in the security model and the worker docs.
Now, portable: a dedicated workspace volume. Task workspaces (sisyphus-task-*, sisyphus-input-*) are created with os.MkdirTemp("", …) in packages/runtime/container.go, so they already follow TMPDIR. An explicit worker flag such as --task-dir, documented as "put this on its own volume or quota'd mount", would give operators a real hard bound on any OS. A runaway task then fills that volume rather than the worker's system disk. It's per-worker, not per-task, but it's honest and enforced by the filesystem. The worker should also turn ENOSPC into a clean task failure rather than a crash.
Later: a Linux quota-backed per-task workspace. Advertise it only after a positive capability check, and fail closed for work that requires it. That's worth doing when hardened work needs per-task isolation from other tasks on the same worker.

If you agree, I'd suggest you take 2 (flag, ENOSPC handling, docs) and I'll write 1 for the security model, since that's where I've been working. Or you take both. Happy either way.

Thanks, that's a clear analysis, and I agree a watcher shouldn't be called a bound. My preference is the second option now, plus one cheap portable step, with the Linux quota capability later. 1. **Now: state the limitation plainly.** Hardened tasks are confined in what they can read and in their network access, but not in how much disk they can write while running. `/output` is checked only after exit. This goes in the security model and the worker docs. 2. **Now, portable: a dedicated workspace volume.** Task workspaces (`sisyphus-task-*`, `sisyphus-input-*`) are created with `os.MkdirTemp("", …)` in `packages/runtime/container.go`, so they already follow `TMPDIR`. An explicit worker flag such as `--task-dir`, documented as "put this on its own volume or quota'd mount", would give operators a real hard bound on any OS. A runaway task then fills that volume rather than the worker's system disk. It's per-worker, not per-task, but it's honest and enforced by the filesystem. The worker should also turn ENOSPC into a clean task failure rather than a crash. 3. **Later: a Linux quota-backed per-task workspace.** Advertise it only after a positive capability check, and fail closed for work that requires it. That's worth doing when hardened work needs per-task isolation from other tasks on the same worker. If you agree, I'd suggest you take 2 (flag, ENOSPC handling, docs) and I'll write 1 for the security model, since that's where I've been working. Or you take both. Happy either way.

excho0 commented

2026-10-20 07:21:07 +00:00

I agree with that split. I’m taking option 2: --task-dir for task workspaces and shared input staging, requiring an existing writable directory so a missing volume cannot silently fall back to the system disk, and handling filesystem-full errors as task failures. I’ll update the worker documentation to say plainly that /output is checked after exit and that the flag only contains disk use when the operator places it on a dedicated/quota-backed filesystem; it is not a per-task quota. Please take the security-model wording as proposed. I’ll request you as reviewer on the daemon PR. /cc @rvnminers-A-and-N

I agree with that split. I’m taking option 2: `--task-dir` for task workspaces and shared input staging, requiring an existing writable directory so a missing volume cannot silently fall back to the system disk, and handling filesystem-full errors as task failures. I’ll update the worker documentation to say plainly that `/output` is checked after exit and that the flag only contains disk use when the operator places it on a dedicated/quota-backed filesystem; it is not a per-task quota. Please take the security-model wording as proposed. I’ll request you as reviewer on the daemon PR. /cc @rvnminers-A-and-N

rvnminers-A-and-N commented

2026-10-20 07:24:02 +00:00

Thanks, that sounds right, including requiring an existing directory so a missing volume can't fall back to the system disk. The security-model half is up as #560. It points owners at TMPDIR for now, so your PR can switch that advice to --task-dir when it lands.

Thanks, that sounds right, including requiring an existing directory so a missing volume can't fall back to the system disk. The security-model half is up as #560. It points owners at `TMPDIR` for now, so your PR can switch that advice to `--task-dir` when it lands.

rvnminers-A-and-N commented

2026-10-20 08:31:38 +00:00

Design question for the "later" per-task quota item: Discussion #565 (project quotas versus a loop image, how to get the privilege, and how a worker advertises the capability).

excho0 commented

2026-10-28 09:47:00 +00:00

I added a source-based connectivity status note in PR #743: open-network-connectivity.md. It distinguishes the DHT and mDNS discovery that are already wired, opportunistic hole punching, Sisyphus-managed relay fallback, and location lookup from actual reachability. It also calls out the cold-start gap: --bootstrap is a configurable mechanism, but there is no public bootstrap deployment plan. The real two-router/NAT checks and sandbox evaluation remain outstanding, so I have not checked those items off. /cc @rvnminers-A-and-N

I added a source-based connectivity status note in PR #743: [open-network-connectivity.md](https://git.echelonts.net/Sisyphus/Sisyphus/src/branch/docs/open-network-connectivity/docs/research/open-network-connectivity.md). It distinguishes the DHT and mDNS discovery that are already wired, opportunistic hole punching, Sisyphus-managed relay fallback, and location lookup from actual reachability. It also calls out the cold-start gap: `--bootstrap` is a configurable mechanism, but there is no public bootstrap deployment plan. The real two-router/NAT checks and sandbox evaluation remain outstanding, so I have not checked those items off. /cc @rvnminers-A-and-N

excho0 commented

2026-10-28 10:42:19 +00:00

The source-based connectivity note from PR #743 is now merged as e00acf0. It separates DHT/mDNS discovery, opportunistic hole punching, relay fallback, and location lookup from verified reachability, and records the cold-start bootstrap limitation. This is documentation/research progress only; I have left the discovery/NAT and sandbox acceptance items open because they still need design and real-machine validation. /cc @rvnminers-A-and-N

The source-based connectivity note from PR #743 is now merged as `e00acf0`. It separates DHT/mDNS discovery, opportunistic hole punching, relay fallback, and location lookup from verified reachability, and records the cold-start bootstrap limitation. This is documentation/research progress only; I have left the discovery/NAT and sandbox acceptance items open because they still need design and real-machine validation. /cc @rvnminers-A-and-N

excho0 commented

2026-10-28 20:17:57 +00:00

Backlog audit catch-up: #306’s abuse-boundary research note was merged, but #22 did not record it. I linked it under Progress and marked only the documentation/research slice complete. The abuse-policy checkbox remains open: reporting, enforcement, acceptable-work rules and resource budgets still need decisions. /cc @rvnminers-A-and-N

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Sisyphus/Sisyphus#22