Storage backends: IPFS/IPLD/IPNS and S3-compatible stores (Ceph and others) #25

New Issue

2026-10-05T05:27:21Z

rvnminers-A-and-N commented

2026-10-05 05:27:21 +00:00

Proposal to build artifact storage (#11) around pluggable backends, with IPFS-family technology as the network-wide layer and S3-compatible storage for pools that run their own.

This changes something in the README, which currently lists IPFS under what V0 deliberately excludes. Needs agreement from both of us: either IPFS moves into V0, or V0 ships with local storage only and the interface is designed so IPFS drops in afterwards. The second is the lower-risk option.

Shape

One small interface in the daemon (put, get, has, delete by content hash), several implementations:

Backend	Covers	Notes
Local disk	Single machines, tests	Needed regardless; first to build.
IPFS (Kubo)	Public, content-addressed sharing between strangers	Workers fetch inputs by CID from whoever has them.
S3-compatible	Ceph (through its RADOS Gateway), MinIO, SeaweedFS, Storj's gateway, AWS S3	One integration reaches all of these. Suits private pools and large datasets.

Identify every artifact by an IPFS-style CID from day one, even on local disk, so the same identifier works on every backend.

Where each IPFS-family piece fits

IPFS: moving job inputs and outputs between nodes that do not know each other.
IPLD: representing task graphs, job manifests and result receipts as linked, hash-addressed data. A job's whole history becomes one root hash, which is what a chain would later commit to.
IPNS: stable names for things that change, such as a node's current capability record or the latest version of a workload manifest. Updates are slow to propagate, so not for anything time-sensitive.
libp2p: peer discovery, NAT traversal and relays for the open network (#22). Probably the most valuable piece of the family for us.

Problems to design for

IPFS does not keep data alive on its own. Data survives only while some node pins it, so we must decide who pins job data, for how long, and who pays.
Everything on public IPFS is readable by anyone who has the CID. Private job data needs encryption before it leaves the requester.
IPFS is slow for very large or very numerous files compared with direct transfer; big datasets may need S3-style storage or direct node-to-node streaming.
Ceph is infrastructure an operator runs, not a network strangers join. It fits "a pool brings its own storage", not network-wide sharing.

Tasks

Agree whether IPFS is in V0 or immediately after, and update the README.
Define the storage interface as part of #11.
Decide on CID format and chunking so local, IPFS and S3 backends agree on identifiers.
Prototype: embed Kubo/Boxo as a library versus talking to a separate Kubo daemon over its RPC API.
Prototype the S3 backend against MinIO, then verify against Ceph RGW.
Decide which other decentralised stores, if any, are worth native support: Storj, Sia, Swarm, Filecoin, Arweave.
Pinning and retention policy.
Encryption of private artifacts.

Proposal to build artifact storage (#11) around pluggable backends, with IPFS-family technology as the network-wide layer and S3-compatible storage for pools that run their own. **This changes something in the README**, which currently lists IPFS under what V0 deliberately excludes. Needs agreement from both of us: either IPFS moves into V0, or V0 ships with local storage only and the interface is designed so IPFS drops in afterwards. The second is the lower-risk option. ## Shape One small interface in the daemon (put, get, has, delete by content hash), several implementations: | Backend | Covers | Notes | | --- | --- | --- | | Local disk | Single machines, tests | Needed regardless; first to build. | | IPFS (Kubo) | Public, content-addressed sharing between strangers | Workers fetch inputs by CID from whoever has them. | | S3-compatible | Ceph (through its RADOS Gateway), MinIO, SeaweedFS, Storj's gateway, AWS S3 | One integration reaches all of these. Suits private pools and large datasets. | Identify every artifact by an IPFS-style CID from day one, even on local disk, so the same identifier works on every backend. ## Where each IPFS-family piece fits - **IPFS**: moving job inputs and outputs between nodes that do not know each other. - **IPLD**: representing task graphs, job manifests and result receipts as linked, hash-addressed data. A job's whole history becomes one root hash, which is what a chain would later commit to. - **IPNS**: stable names for things that change, such as a node's current capability record or the latest version of a workload manifest. Updates are slow to propagate, so not for anything time-sensitive. - **libp2p**: peer discovery, NAT traversal and relays for the open network (#22). Probably the most valuable piece of the family for us. ## Problems to design for - IPFS does not keep data alive on its own. Data survives only while some node pins it, so we must decide who pins job data, for how long, and who pays. - Everything on public IPFS is readable by anyone who has the CID. Private job data needs encryption before it leaves the requester. - IPFS is slow for very large or very numerous files compared with direct transfer; big datasets may need S3-style storage or direct node-to-node streaming. - Ceph is infrastructure an operator runs, not a network strangers join. It fits "a pool brings its own storage", not network-wide sharing. ## Tasks - [ ] Agree whether IPFS is in V0 or immediately after, and update the README. - [ ] Define the storage interface as part of #11. - [ ] Decide on CID format and chunking so local, IPFS and S3 backends agree on identifiers. - [ ] Prototype: embed Kubo/Boxo as a library versus talking to a separate Kubo daemon over its RPC API. - [ ] Prototype the S3 backend against MinIO, then verify against Ceph RGW. - [ ] Decide which other decentralised stores, if any, are worth native support: Storj, Sia, Swarm, Filecoin, Arweave. - [ ] Pinning and retention policy. - [ ] Encryption of private artifacts.

rvnminers-A-and-N commented

2026-10-05 05:50:09 +00:00

Persistence, privacy, and where Ceph fits: options to discuss

#26 implements the local-disk part of this proposal with real IPFS CIDs. What follows is the design space for the two hard questions, with a suggested default for each. Nothing here is decided.

Why Ceph and IPFS feel similar, and how they differ

Both spread data over many machines. The difference is who is trusted.

	Ceph	IPFS
Who runs it	One operator, machines that trust each other	Anyone; peers do not trust each other
How data is named	By location (pool, bucket, key)	By content hash (CID)
Keeps data alive	Yes: replication or erasure coding, self-healing	Only while some peer chooses to pin it
Mutable	Yes	No; a change is a new CID
Strangers can join	No	Yes

So they stack rather than compete. Ceph, or anything else speaking S3 (MinIO, SeaweedFS, Storj's gateway, Wasabi, AWS), is durable storage one party operates. IPFS is how untrusting parties name and exchange data. A Kubo node can keep its blocks in an S3 bucket through the go-ds-s3 datastore plugin, so a pool with a Ceph cluster can serve IPFS content out of it.

Persistence: who keeps the bytes, and for how long

Tier	Who holds the data	Survives	Cost	When
1. Job-scoped	Requester's node holds inputs; workers hold outputs until the requester's node confirms it has them, then a short expiry	Nothing beyond the job	Free	V0
2. Pool-pinned	The pool's own storage: IPFS Cluster with a replication factor, or an S3/Ceph bucket	Loss of individual nodes	Pool operator's hardware	After V0
3. Network-pinned	Storage providers on the open network, paid in SISY	Requester and workers going offline	Paid per size and duration	Needs the chain
4. Archival	An outside network or service: Filecoin, Arweave, Storj, Sia, S3	Sisyphus itself	That service's price	Optional

Notes on each:

Tier 1 is enough for V0 and needs no IPFS networking, only what #26 has plus transfer between nodes.
Tier 2 with IPFS Cluster. Cluster runs beside Kubo on each storage node and keeps a shared list of pinned CIDs, with a minimum and maximum replication factor per pin, re-pinning elsewhere when a node drops out. It also has follower peers that replicate without being able to change the pin list, which suits volunteers donating disk.
Tier 3 is its own research problem. Content addressing makes a cheap spot check possible: ask a provider for a randomly chosen block of a CID and verify the hash. That shows the provider had that block at that moment. It does not show that the data is stored several times independently or will still be there tomorrow; Filecoin's proof systems exist because that is hard. A realistic first version is spot checks plus collateral that is forfeited on failure.
Chain tie-in. Evrmore's asset layer, inherited from Ravencoin, can attach an IPFS hash to an asset. A finished job's result root could be recorded that way, which is one concrete reason to want the asset layer (#20).

Suggested default: tier 1 in V0; design the retention field on a blob now (expiry time, or "pinned until released") so tiers 2 and 3 are a policy change rather than a redesign.

Privacy: who can read the bytes

Tier	Hidden from	Not hidden from	How
A. Public	Nobody	Everyone	Plain data, public CIDs
B. Private pool	Everyone outside the pool	Every pool member	Private IPFS swarm (shared swarm key, no public bootstrap peers, CIDs never announced to the public DHT) plus TLS between nodes (#8)
C. Encrypted artifacts	Storage nodes, relays, bystanders	The workers assigned to the job	Requester encrypts each job's data with a fresh random key; the key is wrapped to each assigned worker's public key
D. Confidential compute	The worker's operator as well	The hardware vendor	Trusted execution environments

Points that are easy to get wrong:

A worker has to see the data to compute on it. Tiers A to C do not change that. If the data must be hidden from whoever runs the hardware, the options are a pool of machines you trust (B) or trusted hardware (D). Computing on encrypted data directly is far too slow for these workloads.
Tier D is research. GPU support in trusted execution is limited and vendor-specific. Track it; do not plan around it.
Use a random key per job, not a key derived from the content. Content-derived keys let identical files deduplicate, but they also let anyone holding a file confirm that you stored it.
Encryption does not hide metadata. Sizes, timing, and who fetched which CID remain visible. Announcing a private CID to the public DHT publishes who has it.
Nothing published can be recalled. The practical form of deletion is destroying the key.
A private swarm is all or nothing. Everyone holding the swarm key can fetch everything in that swarm. It separates a pool from the world, not pool members from each other.

Suggested default: B for private pools, since V0 is trusted machines anyway; design C alongside node identity in #8, because it needs each node to have a keypair; state plainly in the docs that public workers can read what they compute on.

Questions for us to settle

Is tier 1 persistence acceptable for V0, with tiers 2 and 3 later?
Does a private pool run its own private IPFS swarm, or just direct node-to-node transfer over TLS? The second is much less to build and gives the same privacy; the swarm matters once pools are large or data is shared between jobs.
Do we want a storage-provider role on the network at all (tier 3), or leave long-term storage to existing networks (tier 4)?
Which outside stores get first-class support beyond generic S3?
Is "public workers can read your data" acceptable for the open network's first version?

## Persistence, privacy, and where Ceph fits: options to discuss #26 implements the local-disk part of this proposal with real IPFS CIDs. What follows is the design space for the two hard questions, with a suggested default for each. Nothing here is decided. ### Why Ceph and IPFS feel similar, and how they differ Both spread data over many machines. The difference is who is trusted. | | Ceph | IPFS | | --- | --- | --- | | Who runs it | One operator, machines that trust each other | Anyone; peers do not trust each other | | How data is named | By location (pool, bucket, key) | By content hash (CID) | | Keeps data alive | Yes: replication or erasure coding, self-healing | Only while some peer chooses to pin it | | Mutable | Yes | No; a change is a new CID | | Strangers can join | No | Yes | So they stack rather than compete. Ceph, or anything else speaking S3 (MinIO, SeaweedFS, Storj's gateway, Wasabi, AWS), is **durable storage one party operates**. IPFS is **how untrusting parties name and exchange data**. A Kubo node can keep its blocks in an S3 bucket through the `go-ds-s3` datastore plugin, so a pool with a Ceph cluster can serve IPFS content out of it. ### Persistence: who keeps the bytes, and for how long | Tier | Who holds the data | Survives | Cost | When | | --- | --- | --- | --- | --- | | 1. Job-scoped | Requester's node holds inputs; workers hold outputs until the requester's node confirms it has them, then a short expiry | Nothing beyond the job | Free | V0 | | 2. Pool-pinned | The pool's own storage: IPFS Cluster with a replication factor, or an S3/Ceph bucket | Loss of individual nodes | Pool operator's hardware | After V0 | | 3. Network-pinned | Storage providers on the open network, paid in SISY | Requester and workers going offline | Paid per size and duration | Needs the chain | | 4. Archival | An outside network or service: Filecoin, Arweave, Storj, Sia, S3 | Sisyphus itself | That service's price | Optional | Notes on each: - **Tier 1 is enough for V0** and needs no IPFS networking, only what #26 has plus transfer between nodes. - **Tier 2 with IPFS Cluster.** Cluster runs beside Kubo on each storage node and keeps a shared list of pinned CIDs, with a minimum and maximum replication factor per pin, re-pinning elsewhere when a node drops out. It also has follower peers that replicate without being able to change the pin list, which suits volunteers donating disk. - **Tier 3 is its own research problem.** Content addressing makes a cheap spot check possible: ask a provider for a randomly chosen block of a CID and verify the hash. That shows the provider had that block at that moment. It does not show that the data is stored several times independently or will still be there tomorrow; Filecoin's proof systems exist because that is hard. A realistic first version is spot checks plus collateral that is forfeited on failure. - **Chain tie-in.** Evrmore's asset layer, inherited from Ravencoin, can attach an IPFS hash to an asset. A finished job's result root could be recorded that way, which is one concrete reason to want the asset layer (#20). **Suggested default:** tier 1 in V0; design the retention field on a blob now (expiry time, or "pinned until released") so tiers 2 and 3 are a policy change rather than a redesign. ### Privacy: who can read the bytes | Tier | Hidden from | Not hidden from | How | | --- | --- | --- | --- | | A. Public | Nobody | Everyone | Plain data, public CIDs | | B. Private pool | Everyone outside the pool | Every pool member | Private IPFS swarm (shared swarm key, no public bootstrap peers, CIDs never announced to the public DHT) plus TLS between nodes (#8) | | C. Encrypted artifacts | Storage nodes, relays, bystanders | The workers assigned to the job | Requester encrypts each job's data with a fresh random key; the key is wrapped to each assigned worker's public key | | D. Confidential compute | The worker's operator as well | The hardware vendor | Trusted execution environments | Points that are easy to get wrong: - **A worker has to see the data to compute on it.** Tiers A to C do not change that. If the data must be hidden from whoever runs the hardware, the options are a pool of machines you trust (B) or trusted hardware (D). Computing on encrypted data directly is far too slow for these workloads. - **Tier D is research.** GPU support in trusted execution is limited and vendor-specific. Track it; do not plan around it. - **Use a random key per job, not a key derived from the content.** Content-derived keys let identical files deduplicate, but they also let anyone holding a file confirm that you stored it. - **Encryption does not hide metadata.** Sizes, timing, and who fetched which CID remain visible. Announcing a private CID to the public DHT publishes who has it. - **Nothing published can be recalled.** The practical form of deletion is destroying the key. - **A private swarm is all or nothing.** Everyone holding the swarm key can fetch everything in that swarm. It separates a pool from the world, not pool members from each other. **Suggested default:** B for private pools, since V0 is trusted machines anyway; design C alongside node identity in #8, because it needs each node to have a keypair; state plainly in the docs that public workers can read what they compute on. ### Questions for us to settle - [ ] Is tier 1 persistence acceptable for V0, with tiers 2 and 3 later? - [ ] Does a private pool run its own private IPFS swarm, or just direct node-to-node transfer over TLS? The second is much less to build and gives the same privacy; the swarm matters once pools are large or data is shared between jobs. - [ ] Do we want a storage-provider role on the network at all (tier 3), or leave long-term storage to existing networks (tier 4)? - [ ] Which outside stores get first-class support beyond generic S3? - [ ] Is "public workers can read your data" acceptable for the open network's first version?

rvnminers-A-and-N commented

2026-10-05 06:50:22 +00:00

Where I (rvnminers-A-and-N) stand on the three questions, for discussion

On each I want at least the middle option, and would rather go all the way.

Question	Minimum	Preferred
Persistence	Job-scoped by default, plus "keep until released" and a record of which blobs belong to which job	Full pinning with retention, replication across a pool, and network-level storage
Transfer in private pools	Direct transfer, with workers also serving blobs to each other	A full private swarm
Long-term storage	Masternodes required to pin recent results for a fixed window	Our own paid storage-provider role, since this becomes its own chain

I also want the IPFS family used deeply (IPLD, IPNS, gateways, pinning services, Cluster), run on our own infrastructure by default, with public gateways and pinning services as an opt-in for people who are fine with them.

These are my views, not decisions; nothing below is built yet and all of it is open to change once we have talked.

Proposed build order

Each step is useful on its own and none blocks V0's exit.

#28 Pins, retention, job-to-blob records, garbage collection
#29 Workers serve blobs to each other
#30 Worker cache that survives restarts
#31 Private swarms over libp2p
#32 Pool-level replication with IPFS Cluster
#33 S3-compatible backend (Ceph, MinIO, Storj, Wasabi)
#34 IPNS, gateways, pinning services, IPLD job records
#35 Storage provider role (needs the chain)
#36 Asset fields for CIDs and retrieval hints (part of the chain design)

Steps 1 to 3 extend what #26 and #27 already do. Step 4 onward adds new networking and should wait for node identity and authentication (#8).

Still to decide together

Whether IPFS moves into V0 in the README, or stays "immediately after".
Whether steps 4 to 7 belong before or after the first multi-machine demo (#18).
The strict coverage rule added in #27.

## Where I (rvnminers-A-and-N) stand on the three questions, for discussion On each I want at least the middle option, and would rather go all the way. | Question | Minimum | Preferred | | --- | --- | --- | | Persistence | Job-scoped by default, plus "keep until released" and a record of which blobs belong to which job | Full pinning with retention, replication across a pool, and network-level storage | | Transfer in private pools | Direct transfer, with workers also serving blobs to each other | A full private swarm | | Long-term storage | Masternodes required to pin recent results for a fixed window | Our own paid storage-provider role, since this becomes its own chain | I also want the IPFS family used deeply (IPLD, IPNS, gateways, pinning services, Cluster), run on our own infrastructure by default, with public gateways and pinning services as an opt-in for people who are fine with them. These are my views, not decisions; nothing below is built yet and all of it is open to change once we have talked. ### Proposed build order Each step is useful on its own and none blocks V0's exit. 1. #28 Pins, retention, job-to-blob records, garbage collection 2. #29 Workers serve blobs to each other 3. #30 Worker cache that survives restarts 4. #31 Private swarms over libp2p 5. #32 Pool-level replication with IPFS Cluster 6. #33 S3-compatible backend (Ceph, MinIO, Storj, Wasabi) 7. #34 IPNS, gateways, pinning services, IPLD job records 8. #35 Storage provider role (needs the chain) 9. #36 Asset fields for CIDs and retrieval hints (part of the chain design) Steps 1 to 3 extend what #26 and #27 already do. Step 4 onward adds new networking and should wait for node identity and authentication (#8). ### Still to decide together - Whether IPFS moves into V0 in the README, or stays "immediately after". - Whether steps 4 to 7 belong before or after the first multi-machine demo (#18). - The strict coverage rule added in #27.

rvnminers-A-and-N commented

2026-10-09 08:32:28 +00:00

Settled and built: content IDs everywhere, one store interface with disk, Kubo (as a sidecar, #43) and S3 (#114) behind it, pins and retention (#38), sealing of private data (#47). The one question left open here, which other decentralised stores deserve native support (Storj, Sia, Swarm, Filecoin, Arweave), has no work behind it; Storj is reachable today through its S3 gateway.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Sisyphus/Sisyphus#25