feat: a coordinator that lost its store takes back what it signed for (#125)

A coordinator now signs what it tells each storage follower to hold: the
blobs, until when each is kept, the store's name and a number that only
goes up. The follower keeps the last list beside its copies. A
coordinator that comes back with its key and an empty store asks to be
shown the list, checks its own signature, fetches each blob named from
that follower and pins it as "restored" until the expiry the list gave.
A list that was changed, or that another key signed, is ignored and
logged. Nothing comes back on a follower's word.

"blob restore" stays as the manual way, for what cannot be checked. A
coordinator that lost its key too is started with --formerly <old ID>,
so that what followers hold for the old node is kept rather than
dropped. "blob replicas" says what has come back and what has yet to,
and "blob unpin" releases a restored blob.

apps/sisyphusd/README.md

@@ -282,6 +282,7 @@ A node keeps a blob for as long as something pins it, and deletes what nothing p
 | Pin held for | Placed by | Lasts |
 | --- | --- | --- |
 | `user` | `blob put` and `blob pin` | Until `blob unpin`, or for `--ttl` |
+| `restored` | A coordinator that lost its store, on what it took back from its followers | Until `blob unpin`, or until the time the lost store kept it to; see [Copies on other nodes](#copies-on-other-nodes) |
 | `job:<id>` | The coordinator, on a job's inputs and results | While the job runs, then for `--retain` (default 7 days) |
 | `record:<id>` | The coordinator, on a finished job's [record](#a-jobs-record) | Until the job is forgotten (`--keep-jobs`) |
 | `recent` | Every upload | One hour, so there is time to pin it properly |
@@ -392,6 +393,7 @@ bin/sisyphusd blob replicas <cid>
 
 - **What is copied** is every blob the coordinator has pinned for longer than the hour a new upload gets: what `blob put` and `blob pin` keep, and what jobs keep. When the last such pin on a blob is released or lapses, the followers delete their copies. The record of a job is not copied: it is linked data and not a file, and can be made again from the job.
 - **A follower decides nothing.** Once a minute it asks the coordinator what to hold and says what it holds. It fetches what it lacks from the coordinator, checked against its CID, and deletes what is no longer listed. No other node can hand it anything to keep, and nothing it says changes what the coordinator keeps.
+- **The coordinator signs each list**, and the follower keeps the last one beside its copies. That is what lets a coordinator that has lost its store take its data back without trusting anyone; see [below](#when-the-coordinators-store-is-lost).
 - **The copies are in a store of their own**, in `--replica-dir`, written through to the disk like a coordinator's and apart from the worker's cache. Use the directory for nothing else: whatever is in it that the coordinator has not listed is deleted. It is always on the follower's disk, whether or not the pool uses `--kubo` or a bucket.
 - **Which followers hold a blob** is settled by hashing each follower's ID with the blob's, so every blob ranks the followers in an order of its own and the first few hold it. Data therefore spreads evenly, and a follower that joins takes a share from each of the others while nothing moves between them.
 - **A copy that is to move stays where it is** until the follower taking it over says it has it. Changing the followers never leaves fewer copies than there were.
@@ -417,30 +419,51 @@ bin/sisyphusd blob replicas <cid>
 
 #### When the coordinator's store is lost
 
-A coordinator knows its store by a name kept beside the store's pins. If it comes back with its key and an empty store, because the disk was replaced, the name is new, and the followers are not told to delete what they hold for the old one.
+Start the coordinator again with the same key. It takes back what its followers hold, by itself.
 
 ```sh
-bin/sisyphusd blob replicas         # says how many blobs the followers hold from the store that was lost
+bin/sisyphusd run --name alpha --replicas 2    # the same data directory, with node.key in it and a new, empty blobs/
-bin/sisyphusd blob get <cid>        # a blob that is asked for is fetched back from a follower then and there
+bin/sisyphusd blob replicas                    # says what has come back, and what has yet to
-bin/sisyphusd blob restore          # fetch all of it back, and pin it
+bin/sisyphusd blob pins                        # what came back is held for "restored"
 ```
 
-- `blob restore` pins what it fetches for `user`, until unpinned. Who had pinned it before, and for how long, was known only to the store that was lost.
+- **The coordinator signs what it tells each follower to hold.** Each answer to a follower is a list: the blobs, until when each is kept, the name of the store, and a number that only goes up, signed with the node's key. The follower keeps the last list in `kept.list` in its `--replica-dir`, beside the copies.
-- It acts on its owner's word because it takes the followers' word for what the lost store had pinned.
+- **A store has a name**, kept beside its pins. A coordinator that comes back with its key and an empty store, because the disk was replaced, has a new name. A follower that holds copies for the old one is not told to delete them. It is asked for its list.
-- Once everything a follower holds is pinned again, the follower carries on as before, for the new store.
+- **The coordinator checks its own signature**, and takes the follower's word for nothing. If the list is its own, it fetches each blob named from that follower, checks it against its CID, and pins it. A list that was changed, or that another key signed, is ignored and logged, and nothing is taken back from it.
+- **What comes back is pinned for `restored`**, until the time the list gave: the latest of the pins the lost store had on the blob. What had no expiry has none. What has since run out is not taken back, and the follower deletes it. Who had pinned each blob is not in the list. `blob unpin` releases a restored blob as it does one pinned by hand.
+- **Several followers each show a list.** Each is taken. A blob that two lists name is kept until the later of the two times.
+- **It starts when the follower next asks**, within a minute of the follower finding the coordinator again, and is done when everything named has been fetched. What cannot be fetched is tried again each minute. `blob replicas` says how many blobs have come back and how many are still to come.
+- **A blob that is asked for** before then is fetched back from a follower there and then, as it always was.
 - To give the old data up instead, stop each follower and empty its `--replica-dir`.
-- For some seconds after the coordinator starts, its followers are not yet reachable from it, and a `blob get` may still say the blob is not found.
+
+##### When `blob restore` is needed
+
+`sisyphusd blob restore` fetches back everything the followers hold from an earlier store, on the followers' word, and pins it for `user` until unpinned. With the key, it is not needed. It is for what the coordinator cannot check:
+
+- **The key was lost too.** A new key makes a new node, and a list signed with the old key cannot be told from a forgery. Start the new coordinator with `--formerly <the old node ID>`, so that it keeps what followers hold for the old node instead of having them delete it. On each follower's machine run `sisyphusd pool join --addr <coordinator> <invitation>` with an invitation from the new coordinator, and start the follower again with the same `--replica-dir`. Then run `blob restore`. The old ID is in `pins.json` in each follower's `--replica-dir`: every pin there is owned by `pool:<the old node ID>/<the old store's name>`.
+- **A follower has no list**: it last heard from a coordinator older than signed lists, or its `kept.list` is gone or damaged.
+- **A follower holds blobs its list does not name.** While a follower is still holding for an earlier store, because something of it has yet to come back, it is given no new list, and what it is handed in that time is on none. If the store is lost again before that is over, those blobs are kept on the follower and not taken back.
+
+`blob replicas` says which case applies: blobs "that it has yet to take back" need nothing, and blobs "that no list signed by this node names" need `blob restore`. Run when there is nothing of the kind, it says so and does nothing.
+
+##### What a follower can still do
+
+- **It cannot add to what is kept.** Only what the coordinator pinned is ever in a list it signed. Blobs from an earlier store that no signed list names are never in one.
+- **It can show an old list.** A follower that kept a list from before a pin was released can show it to a coordinator that has lost its store, and the blob comes back, until the expiry that list gave it, or until unpinned if it had none. That brings back only what the coordinator once chose to keep, and it needs the coordinator to have lost its store: a list of the store it has now brings nothing back.
+- **A blob taken back and then released stays released** while the coordinator runs. A coordinator that is started again while a follower still shows a list of the lost store takes it back once more.
 
 #### What this does not protect
 
-- **The coordinator's database and key.** `node.db` holds the jobs, the members and the invitations, and `node.key` is who the node is: followers know their coordinator by it. Neither is copied anywhere. Back them up; without the key a new coordinator is a different node, which the followers will not answer.
+- **The coordinator's database and key.** `node.db` holds the jobs, the members and the invitations, and `node.key` is who the node is: followers know their coordinator by it, and it is what makes a list the coordinator's own. Neither is copied anywhere. Back them up. Without the key, getting the data back takes `--formerly`, a new invitation for every follower, and `blob restore`, and the expiries are lost.
-- **Who pinned what, and until when.** Followers hold blobs, not the pins on them.
+- **Who pinned what.** Followers hold blobs and, in the list, until when each is kept. A blob a job was holding while it ran comes back with no expiry, and stays until `blob unpin`.
 - **What is not pinned**, and what was pinned in the last minute or so, before a follower had asked and fetched it.
 - **A follower is taken at its word** for what it holds. Nothing checks later that the copy is still there and sound; a damaged one is found out only when it is fetched and fails its check.
 - **A follower can read what it holds**, as any worker can fetch any blob. Private jobs seal their data, and the copies are of the sealed form.
 - There is one number of copies for the whole pool, not one per pin, and copies are whole: no erasure coding.
 - The intervals are fixed: a minute between asks, ten minutes before a follower is given up on.
-- The list of what to hold travels whole each time, which bounds a pool at about sixty thousand pinned blobs.
+- The list of what to hold travels whole each time, which bounds a pool at about sixty thousand pinned blobs. A follower that shows its list sends it along with what it holds, so a follower that holds more than about thirty thousand blobs of a lost store cannot show its list: it keeps what it has, and `blob restore` takes it back.
+- A coordinator takes a list back while it answers the follower that showed it. That follower hears nothing new until it is done.
+
 ### Pinning services
 
 The [IPFS Pinning Service API](https://ipfs.github.io/pinning-services-api-spec/) is how one machine asks another, over HTTP, to keep data by its CID. A node speaks it both ways: it can be the service, and it can ask one.
@@ -686,7 +709,7 @@ A coordinator keeps its jobs in `node.db`, a SQLite database in its data directo
 - **Unfinished jobs** are taken up where they were. Tasks that had finished stay finished. Tasks that were running go back to wait for a worker, and that is not counted against them, however often it happens. If every task had finished and only combining their outputs was cut short, the combining is done again, with no worker needed.
 - **A job's data** stays pinned across the restart for as long as the job takes.
 - **Workers** reconnect by themselves once the coordinator is back.
-- **Storage followers** go on holding what they held. See [Copies on other nodes](#copies-on-other-nodes) for a coordinator that comes back without its stored data.
+- **Storage followers** go on holding what they held. A coordinator that comes back without its stored data takes it back from them; see [Copies on other nodes](#copies-on-other-nodes).
 
 A job's arrival and its finish are written through to the disk before anyone is told of them. The steps in between are not, so a power cut can lose the last few; the tasks concerned run again. A coordinator started without a workload that an unfinished job needs fails that job, and says why.
 
@@ -900,7 +923,7 @@ The video is cut into as many stretches as the job has tasks. Each task encodes
 - A node has one name, its ID, and a record published by hand is not renewed: it stops resolving when its lifetime is over.
 - The records of names reach the pool's private IPFS network only when they are published; a record that has gone from there is not put back until its node publishes again.
 - Without `--max-store-bytes` there is no limit on what a worker or client can upload.
-- Without `--replicas` and at least one follower, a pool's stored data is on its coordinator's disk and nowhere else. With them, the coordinator's database and key are still on that disk alone.
+- Without `--replicas` and at least one follower, a pool's stored data is on its coordinator's disk and nowhere else. With them, the coordinator's database and key are still on that disk alone, and without the key the data comes back only by hand and without its expiries.
 - The pinning service has one key and speaks plain HTTP. Whoever holds the key can fill the store up to `--max-store-bytes`, and a single fetch can go past it.
 - A pool's data can be kept by an outside pinning service only if that service is on the pool's private IPFS network, or has the data from somewhere else.
 - A worker downloads a whole input even when its tasks need only part of it.

apps/sisyphusd/api/blobs.go

@@ -223,12 +223,20 @@ func (s *blobService) Unpin(_ context.Context, req *pb.UnpinBlobRequest) (*pb.Un
 	if err != nil {
 		return nil, status.Errorf(codes.InvalidArgument, "invalid CID %q: %v", req.GetCid(), err)
 	}
-	if err := s.store.Unpin(userOwner, c); err != nil {
+	if err := unpinForUser(s.store, c); err != nil {
 		return nil, status.Errorf(codes.Internal, "unpin blob: %v", err)
 	}
 	return &pb.UnpinBlobResponse{}, nil
 }
 
+// unpinForUser releases the pins that are the node's user's to release: the
+// user's own, and the one on a blob the node took back from its followers
+// after losing its store, which stands in for whatever pins the lost store
+// had on it.
+func unpinForUser(store FileStore, c cid.Cid) error {
+	return errors.Join(store.Unpin(userOwner, c), store.Unpin(replication.RestoredOwner, c))
+}
+
 func (s *blobService) ListPins(context.Context, *pb.ListPinsRequest) (*pb.ListPinsResponse, error) {
 	var res pb.ListPinsResponse
 	for _, pin := range s.store.Pins() {
@@ -254,8 +262,7 @@ func (s *blobService) CollectGarbage(ctx context.Context, _ *pb.CollectGarbageRe
 }
 
 func (s *blobService) Replicate(ctx context.Context, req *pb.ReplicateRequest) (*pb.ReplicateResponse, error) {
-	hold, store := s.replication.Replicate(access.Caller(ctx), req.GetStore(), req.GetHolding())
+	return s.replication.Replicate(ctx, access.Caller(ctx), req), nil
-	return &pb.ReplicateResponse{Hold: hold, Store: store}, nil
 }
 
 func (s *blobService) Replicas(_ context.Context, req *pb.ReplicasRequest) (*pb.ReplicasResponse, error) {

apps/sisyphusd/api/local_files.go

@@ -175,7 +175,7 @@ func (s *localService) RemoveFile(ctx context.Context, req *nodepb.RemoveFileReq
 	if err := s.cfg.Files.RemoveFile(c.String()); err != nil {
 		return nil, status.Errorf(codes.Internal, "%v", err)
 	}
-	if err := s.cfg.Store.Unpin(userOwner, c); err != nil {
+	if err := unpinForUser(s.cfg.Store, c); err != nil {
 		return nil, status.Errorf(codes.Internal, "stop keeping file: %v", err)
 	}
 	return &nodepb.RemoveFileResponse{}, nil

apps/sisyphusd/blob.go

@@ -344,7 +344,8 @@ func blobGC(ctx context.Context, args []string) error {
 }
 
 // blobReplicas shows how many of a node's storage followers hold each blob
-// it has pinned, against how many should.
+// it has pinned, against how many should, and what a node that lost its
+// store has taken back from them or has yet to.
 func blobReplicas(ctx context.Context, args []string) error {
 	fs := flag.NewFlagSet("sisyphusd blob replicas", flag.ContinueOnError)
 	node := targetFlags(fs)
@@ -365,8 +366,19 @@ func blobReplicas(ctx context.Context, args []string) error {
 		return err
 	}
 	wanted, followers := int(listed.GetWanted()), len(listed.GetFollowers())
-	if stranded := listed.GetFromEarlierStore(); stranded > 0 {
+	if listed.GetRestoring() {
-		fmt.Fprintf(stdout, "followers hold %d blob(s) from a store this node had before, which it no longer has pinned: \"sisyphusd blob restore\" fetches them back\n", stranded)
+		fmt.Fprintf(stdout, "this node is taking back what a follower holds from a store it had before: %d blob(s) are back so far\n", listed.GetRestored())
+	} else if restored := listed.GetRestored(); restored > 0 {
+		fmt.Fprintf(stdout, "%d blob(s) are kept that this node took back from its followers after losing its store: \"sisyphusd blob pins\" lists them as held for \"restored\"\n", restored)
+	}
+	// What a list signed by this node names it takes back by itself. The
+	// rest it has only the followers' word for.
+	unsigned := listed.GetUnsignedFromEarlierStore()
+	if signed := listed.GetFromEarlierStore() - unsigned; signed > 0 {
+		fmt.Fprintf(stdout, "followers hold %d blob(s) from a store this node had before that it has yet to take back: it tries again each time a follower asks what to hold, and there is nothing to run\n", signed)
+	}
+	if unsigned > 0 {
+		fmt.Fprintf(stdout, "followers hold %d blob(s) from a store this node had before that no list signed by this node names: it does not take those back by itself, and \"sisyphusd blob restore\" does\n", unsigned)
 	}
 	if wanted == 0 {
 		fmt.Fprintln(stdout, "this node asks no followers to hold copies of its data: it was started without --replicas")
@@ -404,7 +416,8 @@ func blobReplicas(ctx context.Context, args []string) error {
 }
 
 // blobRestore has a node whose store was lost fetch back what its storage
-// followers still hold of it.
+// followers still hold of it, on their word. A node that still has its key
+// does that by itself for whatever it had signed for.
 func blobRestore(ctx context.Context, args []string) error {
 	fs := flag.NewFlagSet("sisyphusd blob restore", flag.ContinueOnError)
 	node := targetFlags(fs)
@@ -425,6 +438,10 @@ func blobRestore(ctx context.Context, args []string) error {
 		return err
 	}
 	fmt.Fprintf(stdout, "restored %d blob(s), pinned until unpinned\n", done.GetRestored())
+	if done.GetRestored() == 0 && len(done.GetFailed()) == 0 {
+		fmt.Fprintln(stdout, "followers hold nothing from an earlier store that this node has not pinned")
+		fmt.Fprintln(stdout, "a node that comes back with its key and without its store takes back by itself what it had signed for; this command is for one that has lost its key too and was started with --formerly, or whose followers kept no signed list")
+	}
 	if failed := done.GetFailed(); len(failed) > 0 {
 		return fmt.Errorf("%d blob(s) could not be fetched back from any follower: %s", len(failed), strings.Join(failed, ", "))
 	}

apps/sisyphusd/daemon.go

@@ -98,6 +98,7 @@ func runDaemon(ctx context.Context, args []string) error {
 	useCluster := fs.Bool("cluster", false, "with --kubo: also run an IPFS Cluster peer beside Kubo, so that what the pool's coordinator pins is kept by several of the pool's nodes; needs the ipfs-cluster-service program installed, and for a worker, a coordinator that uses it too")
 	swarmPort := fs.Int("swarm-port", 0, "coordinator role with --kubo: TCP port to open so that members' Kubo daemons can connect to this node's directly, which is faster; 0 opens none, and they reach it through --listen")
 	replicas := fs.Int("replicas", 0, "coordinator role: how many copies of everything this node has pinned are to be held by other nodes of the pool, besides its own. Without --cluster they are held by storage followers, and 0 asks none to; with --cluster they are held by the cluster's peers, and 0 means 1")
+	formerly := fs.String("formerly", "", "coordinator role: the ID this node had before it lost its key; what storage followers hold for that node is kept until \"sisyphusd blob restore\" takes it back")
 	replicaDir := fs.String("replica-dir", "", "worker-only node: be a storage follower, keeping in this directory, which must be used for nothing else, copies of whatever stored data its coordinator says to")
 	syncCache := fs.Bool("sync-cache", false, "worker-only node: wait for the disk when caching a blob; slower, but the cache then survives a power cut without downloading again")
 	verbose := fs.Bool("v", false, "log per-task detail")
@@ -174,6 +175,14 @@ func runDaemon(ctx context.Context, args []string) error {
 	if *replicas < 0 || (*replicas > 0 && !isCoordinator) {
 		return errors.New("--replicas is a number of copies, for a node that coordinates a pool; a worker-only node offers to hold copies with --replica-dir")
 	}
+	if *formerly != "" {
+		if !isCoordinator {
+			return errors.New("--formerly is for a node that coordinates a pool: it names the node its storage followers held copies for before")
+		}
+		if _, err := identity.Verify(*formerly, nil, nil); err != nil {
+			return fmt.Errorf("--formerly: %w", err)
+		}
+	}
 	if isCoordinator && *replicaDir != "" {
 		return errors.New("--replica-dir is for worker-only nodes; a coordinator holds its pool's data already")
 	}
@@ -407,7 +416,7 @@ func runDaemon(ctx context.Context, args []string) error {
 			followerCopies = 0
 		}
 		replicated := replication.New(replication.Config{
-			ID: ident.ID(), Store: store, StoreID: storeID, Replicas: followerCopies, Log: log,
+			Identity: ident, Store: store, StoreID: storeID, Formerly: *formerly, Replicas: followerCopies, Log: log,
 			Address: func(id string) string { return coord.ServeAddress(id) },
 			// A follower is reached as one worker reaches another: at the
 			// address it gave, or by name through the node's libp2p host.
@@ -573,7 +582,10 @@ func runDaemon(ctx context.Context, args []string) error {
 			}
 			defer copies.Close()
 			served = api.Stores(copies, store)
-			follower := &replication.Follower{Store: copies, Coordinator: pb.NewBlobServiceClient(remote.Conn()), Log: log}
+			// The list the coordinator signed of what to hold is kept with
+			// the copies: it is of no use without them, nor they without it
+			// to a coordinator that has lost its store.
+			follower := &replication.Follower{Store: copies, ListFile: filepath.Join(*replicaDir, "kept.list"), Coordinator: pb.NewBlobServiceClient(remote.Conn()), Log: log}
 			following := make(chan struct{})
 			// The store of copies must outlive whatever fills it.
 			defer func() {

apps/sisyphusd/main.go

@@ -28,13 +28,13 @@ const usage = `Usage:
   sisyphusd blob get <cid>         fetch a blob, checking it against its CID
   sisyphusd blob stat <cid>        show a blob's size
   sisyphusd blob pin <cid>         keep a blob until unpinned, or for --ttl
-  sisyphusd blob unpin <cid>       stop keeping a blob
+  sisyphusd blob unpin <cid>       stop keeping a blob, whether it was pinned by hand or taken back from followers
   sisyphusd blob pins              list what the node is keeping, and for whom
   sisyphusd blob gc                delete stored data that nothing is keeping
   sisyphusd name publish <cid>     point this node's name, which is its ID, at a stored file (--lifetime, --ttl)
   sisyphusd name resolve <node-id> ask a node what a name stands for, check the answer and print the CID
-  sisyphusd blob replicas [<cid>]  show which storage followers hold copies of what the node has pinned
+  sisyphusd blob replicas [<cid>]  show which storage followers hold copies of what the node has pinned, and what it has taken back from them
-  sisyphusd blob restore           fetch back what followers hold from a store this node has lost
+  sisyphusd blob restore           fetch back, on the followers' word, what they hold from a store this node has lost; needed only if it lost its key too
   sisyphusd blob pin-remote <cid> [name]  ask a pinning service to keep a blob too (--service <url> --key-file <file>, --wait)
   sisyphusd blob pins-remote       list what a pinning service has been asked to keep
   sisyphusd blob unpin-remote <request-id>  have a pinning service stop keeping something

apps/sisyphusd/pool_test.go

@@ -80,9 +80,11 @@ type pool struct {
 	// names is where the coordinator keeps the records of names.
 	names *nameShelf
 	// copies are the stores in which the storage followers started by name
-	// keep what they hold for the pool, addresses where each serves blobs,
+	// keep what they hold for the pool, lists the files in which each keeps
+	// the list its coordinator signed, addresses where each serves blobs,
 	// and unfollow how to stop each following while it goes on serving.
 	copies    map[string]*storage.Store
+	lists     map[string]string
 	addresses map[string]string
 	unfollow  map[string]func()
 }
@@ -163,6 +165,13 @@ func startPoolWith(t *testing.T, workloads *runtime.Registry, wrap func(*storage
 // startPoolKeeping is startPoolWith for a pool whose coordinator has that
 // many storage followers hold a copy of whatever it has pinned.
 func startPoolKeeping(t *testing.T, workloads *runtime.Registry, wrap func(*storage.Store) coordinator.Store, journal coordinator.Journal, replicas int) *pool {
+	t.Helper()
+	return startPoolAs(t, workloads, wrap, journal, replication.Config{Identity: newIdentity(t), StoreID: "first", Replicas: replicas})
+}
+
+// startPoolAs is startPoolKeeping for a pool whose coordinator has the key,
+// the name for its store, and the rest that copies says.
+func startPoolAs(t *testing.T, workloads *runtime.Registry, wrap func(*storage.Store) coordinator.Store, journal coordinator.Journal, copies replication.Config) *pool {
 	t.Helper()
 	lis, err := net.Listen("tcp", "127.0.0.1:0")
 	if err != nil {
@@ -170,7 +179,7 @@ func startPoolKeeping(t *testing.T, workloads *runtime.Registry, wrap func(*stor
 	}
 	store := storage.NewMemory()
 	logs := new(syncBuffer)
-	ident := newIdentity(t)
+	ident := copies.Identity
 	// Kept in memory: an access list with no file saves nothing.
 	admitted, err := access.Open(access.InMemory(), ident.ID())
 	if err != nil {
@@ -185,15 +194,13 @@ func startPoolKeeping(t *testing.T, workloads *runtime.Registry, wrap func(*stor
 	}
 	gets := new(atomic.Int32)
 	kept := &nameShelf{records: make(map[string][]byte)}
-	replicated := replication.New(replication.Config{
+	copies.Store, copies.Log, copies.Address = store, slog.New(slog.NewTextHandler(logs, nil)), coord.ServeAddress
-		ID: ident.ID(), Store: store, StoreID: "first", Replicas: replicas, Log: slog.New(slog.NewTextHandler(logs, nil)),
+	copies.Fetch = func(ctx context.Context, from *pb.BlobHolder, c cid.Cid, into blobclient.Putter) error {
-		Address: coord.ServeAddress,
+		return worker.FetchFromPeer(ctx, from, c, into, func(nodeID string) credentials.TransportCredentials {
-		Fetch: func(ctx context.Context, from *pb.BlobHolder, c cid.Cid, into blobclient.Putter) error {
+			return credentials.NewTLS(ident.ClientTLS(nodeID))
-			return worker.FetchFromPeer(ctx, from, c, into, func(nodeID string) credentials.TransportCredentials {
+		}, nil)
-				return credentials.NewTLS(ident.ClientTLS(nodeID))
+	}
-			}, nil)
+	replicated := replication.New(copies)
-		},
-	})
 	srv := api.NewServer(
 		api.Config{Identity: ident, Access: admitted, Coordinator: coord, Store: store, Replication: replicated, Names: api.NewNames(kept, func(id string) bool {
 			_, admitted := admitted.Role(id)
@@ -225,7 +232,8 @@ func startPoolKeeping(t *testing.T, workloads *runtime.Registry, wrap func(*stor
 		ident: ident, access: admitted, workerIdents: make(map[string]*identity.Identity),
 		rawIdents: make(map[*rawWorker]*identity.Identity),
 		names:     kept,
-		copies:    make(map[string]*storage.Store), addresses: make(map[string]string), unfollow: make(map[string]func()),
+		copies:    make(map[string]*storage.Store), lists: make(map[string]string),
+		addresses: make(map[string]string), unfollow: make(map[string]func()),
 	}
 }
 
@@ -233,19 +241,20 @@ func startPoolKeeping(t *testing.T, workloads *runtime.Registry, wrap func(*stor
 // is called, whichever comes first. stop waits for the worker to exit.
 func (p *pool) startWorker(id string, slots int) (stop func()) {
 	p.t.Helper()
-	return p.startMember(id, slots, nil)
+	return p.startMember(id, slots, nil, "")
 }
 
 // startFollower runs a worker that is also a storage follower, as one
 // started with --replica-dir is.
 func (p *pool) startFollower(id string) (stop func()) {
 	p.t.Helper()
-	return p.startMember(id, 1, storage.NewMemory())
+	return p.startMember(id, 1, storage.NewMemory(), filepath.Join(p.t.TempDir(), "kept.list"))
 }
 
-// startMember runs a worker which, given a store for them, also holds
+// startMember runs a worker which, given a store for them and a file for
-// copies of the pool's data as the coordinator directs.
+// the list its coordinator signs, also holds copies of the pool's data as
-func (p *pool) startMember(id string, slots int, copies *storage.Store) (stop func()) {
+// the coordinator directs.
+func (p *pool) startMember(id string, slots int, copies *storage.Store, list string) (stop func()) {
 	p.t.Helper()
 	ctx, cancel := context.WithCancel(context.Background())
 	done := make(chan struct{})
@@ -270,8 +279,8 @@ func (p *pool) startMember(id string, slots int, copies *storage.Store) (stop fu
 	if copies == nil {
 		close(following)
 	} else {
-		p.copies[id], served = copies, api.Stores(copies, local)
+		p.copies[id], p.lists[id], served = copies, list, api.Stores(copies, local)
-		follower := &replication.Follower{Store: copies, Coordinator: pb.NewBlobServiceClient(blobs.Conn()), Log: quiet}
+		follower := &replication.Follower{Store: copies, ListFile: list, Coordinator: pb.NewBlobServiceClient(blobs.Conn()), Log: quiet}
 		asking, unfollow := context.WithCancel(ctx)
 		go func() {
 			defer close(following)

apps/sisyphusd/replication/follower.go

@@ -2,6 +2,7 @@ package replication
 
 import (
 	"context"
+	"fmt"
 	"log/slog"
 	"slices"
 	"strings"
@@ -36,6 +37,10 @@ type Follower struct {
 	// for nothing else: whatever is in it that the coordinator has not
 	// listed is deleted.
 	Store FollowerStore
+	// ListFile is where the last list the coordinator signed is kept. It
+	// should be beside the copies and as durable: it is what a coordinator
+	// that has lost its store takes them back on the strength of.
+	ListFile string
 	// Coordinator is the blob service of the node followed.
 	Coordinator pb.BlobServiceClient
 	Log         *slog.Logger
@@ -86,16 +91,40 @@ func (f *Follower) sync(ctx context.Context) (changed bool, err error) {
 		held[pin.CID] = append(held[pin.CID], pin.Owner)
 		store = name
 	}
-	told, err := f.Coordinator.Replicate(ctx, &pb.ReplicateRequest{Holding: holding, Store: store})
+	asked := &pb.ReplicateRequest{Holding: holding, Store: store, KeepsLists: true}
+	told, err := f.Coordinator.Replicate(ctx, asked)
+	if err == nil && told.GetShowList() {
+		// The coordinator has another store than the one these copies are
+		// of. Shown the list it signed for that one, it can take them back.
+		list, unread := loadList(f.ListFile)
+		if unread != nil {
+			f.Log.Warn("could not read the list this node's coordinator signed of what to hold; without it the coordinator takes nothing back by itself", "error", unread)
+		}
+		if list != nil {
+			asked.List = list
+			told, err = f.Coordinator.Replicate(ctx, asked)
+		}
+	}
 	if err != nil {
 		return false, err
 	}
 
+	hold := told.GetHold()
+	if list := told.GetList(); list != nil {
+		// The list is kept before it is acted on, so that the one on disk
+		// never names less than what is held.
+		if err := saveList(f.ListFile, list); err != nil {
+			return false, fmt.Errorf("keep the list the coordinator signed: %w", err)
+		}
+		for _, blob := range list.GetBlobs() {
+			hold = append(hold, blob.GetCid())
+		}
+	}
 	owner := ownerPrefix + told.GetStore()
 	wanted := make(map[cid.Cid]bool)
 	var pin []cid.Cid
 	failed := false
-	for _, id := range told.GetHold() {
+	for _, id := range hold {
 		c, err := cid.Decode(id)
 		if err == nil && held[c] == nil {
 			// Whatever arrives is checked against its CID as it is stored.

apps/sisyphusd/replication/follower_test.go

@@ -5,6 +5,8 @@ import (
 	"errors"
 	"io"
 	"net"
+	"os"
+	"path/filepath"
 	"slices"
 	"sync"
 	"testing"
@@ -15,6 +17,7 @@ import (
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/credentials/insecure"
 	"google.golang.org/grpc/status"
+	"google.golang.org/protobuf/proto"
 
 	pb "github.com/sisyphus-network/Sisyphus/packages/protocol/sisyphus/v1"
 	"github.com/sisyphus-network/Sisyphus/packages/storage"
@@ -32,6 +35,10 @@ type scripted struct {
 	hold []string
 	name string
 	err  error
+	// list, if set, is the signed list it answers with, and asks is whether
+	// it asks a follower that shows no list for the one it has.
+	list *pb.KeepList
+	asks bool
 	// asked holds every request it has had.
 	asked []*pb.ReplicateRequest
 }
@@ -43,7 +50,7 @@ func (s *scripted) Replicate(_ context.Context, req *pb.ReplicateRequest) (*pb.R
 	if s.err != nil {
 		return nil, s.err
 	}
-	return &pb.ReplicateResponse{Hold: s.hold, Store: s.name}, nil
+	return &pb.ReplicateResponse{Hold: s.hold, Store: s.name, List: s.list, ShowList: s.asks && req.GetList() == nil}, nil
 }
 
 func (s *scripted) Get(req *pb.GetBlobRequest, stream grpc.ServerStreamingServer[pb.GetBlobResponse]) error {
@@ -63,7 +70,19 @@ func (s *scripted) Get(req *pb.GetBlobRequest, stream grpc.ServerStreamingServer
 func (s *scripted) say(name string, err error, hold ...string) {
 	s.mu.Lock()
 	defer s.mu.Unlock()
-	s.name, s.err, s.hold = name, err, hold
+	s.name, s.err, s.hold, s.list = name, err, hold, nil
+}
+
+// sign has the coordinator answer from now on with a list of the given
+// blobs, as one that signs what it says does.
+func (s *scripted) sign(name string, hold ...string) *pb.KeepList {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.name, s.err, s.hold, s.list = name, nil, nil, &pb.KeepList{Store: name, Sequence: uint64(len(s.asked)) + 1, Signature: []byte("signed")}
+	for _, id := range hold {
+		s.list.Blobs = append(s.list.Blobs, &pb.KeptBlob{Cid: id})
+	}
+	return s.list
 }
 
 // requests returns how many times the coordinator has been asked, and the
@@ -96,7 +115,7 @@ func following(t *testing.T) (*scripted, *Follower, *storage.Store, *logged) {
 	}
 	t.Cleanup(func() { conn.Close() })
 	copies, logs := storage.NewMemory(), new(logged)
-	return coordinator, &Follower{Store: copies, Coordinator: pb.NewBlobServiceClient(conn), Log: logs.logger()}, copies, logs
+	return coordinator, &Follower{Store: copies, ListFile: filepath.Join(t.TempDir(), "kept.list"), Coordinator: pb.NewBlobServiceClient(conn), Log: logs.logger()}, copies, logs
 }
 
 // run runs a follower, asking every interval, until the test ends or the
@@ -315,3 +334,112 @@ func TestAFollowerWhoseStoreFailsSaysSoAndDoesNotAskInARush(t *testing.T) {
 		t.Errorf("the follower has asked %d times, want once by hand and once more: a fetch that could not be kept is no reason to ask again at once", asked)
 	}
 }
+
+func TestAFollowerKeepsTheLastListItsCoordinatorSigned(t *testing.T) {
+	coordinator, follower, copies, _ := following(t)
+	first, second := put(t, coordinator.store, "on the first list"), put(t, coordinator.store, "on both lists")
+	if list, err := loadList(follower.ListFile); list != nil || err != nil {
+		t.Fatalf("before it has asked the follower keeps %v, error %v", list, err)
+	}
+
+	signed := coordinator.sign("node/first", first, second)
+	if changed, err := follower.sync(ctx); err != nil || !changed {
+		t.Fatalf("fetching what a list names: changed %v, error %v", changed, err)
+	}
+	want := []string{"pool:node/first " + first, "pool:node/first " + second}
+	slices.Sort(want)
+	if pins := pinsOf(copies); !slices.Equal(pins, want) {
+		t.Errorf("the follower's pins are %v, want the two blobs the list names", pins)
+	}
+	if kept, err := loadList(follower.ListFile); err != nil || !proto.Equal(kept, signed) {
+		t.Errorf("the follower keeps %v, error %v, want the list it was given, signature and all", kept, err)
+	}
+	if _, last := coordinator.requests(); !last.GetKeepsLists() || last.GetList() != nil {
+		t.Errorf("the follower asked with %v, want it to say it keeps lists and to show none unasked", last)
+	}
+
+	// A later list takes the place of the first, and what it leaves out goes.
+	signed = coordinator.sign("node/first", second)
+	if changed, err := follower.sync(ctx); err != nil || !changed {
+		t.Fatalf("dropping what a list no longer names: changed %v, error %v", changed, err)
+	}
+	if kept, err := loadList(follower.ListFile); err != nil || !proto.Equal(kept, signed) {
+		t.Errorf("the follower keeps %v, error %v, want the later list", kept, err)
+	}
+	if holds(t, copies, first) || !holds(t, copies, second) {
+		t.Error("the follower does not hold exactly what the later list names")
+	}
+	// An answer with no list leaves the one it has alone.
+	coordinator.say("node/first", nil, second)
+	if _, err := follower.sync(ctx); err != nil {
+		t.Fatal(err)
+	}
+	if kept, err := loadList(follower.ListFile); err != nil || !proto.Equal(kept, signed) {
+		t.Errorf("after an answer without a list the follower keeps %v, error %v", kept, err)
+	}
+}
+
+func TestAFollowerShowsItsListWhenItsCoordinatorAsks(t *testing.T) {
+	coordinator, follower, copies, logs := following(t)
+	blob := put(t, coordinator.store, "held for a store the coordinator has lost")
+	signed := coordinator.sign("node/first", blob)
+	if _, err := follower.sync(ctx); err != nil {
+		t.Fatal(err)
+	}
+
+	// The coordinator comes back with another store, and asks.
+	coordinator.say("node/first", nil, blob)
+	coordinator.mu.Lock()
+	coordinator.asks = true
+	coordinator.mu.Unlock()
+	before, _ := coordinator.requests()
+	if changed, err := follower.sync(ctx); err != nil || changed {
+		t.Fatalf("showing the list: changed %v, error %v", changed, err)
+	}
+	asked, last := coordinator.requests()
+	if asked != before+2 || !proto.Equal(last.GetList(), signed) || !slices.Equal(last.GetHolding(), []string{blob}) || last.GetStore() != "node/first" {
+		t.Errorf("asked for its list, the follower made %d requests, the last %v, want a second that shows the list it kept", asked-before, last)
+	}
+
+	// A follower whose list cannot be read says so and carries on with what
+	// it was told; one that has none just carries on.
+	if err := os.WriteFile(follower.ListFile, []byte("not a list"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	for _, unreadable := range []int{1, 1} {
+		before, _ = coordinator.requests()
+		if _, err := follower.sync(ctx); err != nil {
+			t.Fatal(err)
+		}
+		if asked, _ := coordinator.requests(); asked != before+1 {
+			t.Errorf("with no list to show the follower made %d requests, want one", asked-before)
+		}
+		if n := logs.count("could not read the list this node's coordinator signed"); n != unreadable {
+			t.Errorf("a list that cannot be read was logged %d times, want %d", n, unreadable)
+		}
+		if err := os.Remove(follower.ListFile); err != nil && !errors.Is(err, os.ErrNotExist) {
+			t.Fatal(err)
+		}
+	}
+	if !holds(t, copies, blob) {
+		t.Error("the follower dropped its copy while its coordinator was asking for a list")
+	}
+}
+
+func TestAFollowerThatCannotKeepAListDoesNotActOnIt(t *testing.T) {
+	coordinator, follower, copies, _ := following(t)
+	held, fresh := put(t, coordinator.store, "held already"), put(t, coordinator.store, "named by a list that cannot be kept")
+	coordinator.say("node/first", nil, held)
+	if _, err := follower.sync(ctx); err != nil {
+		t.Fatal(err)
+	}
+
+	follower.ListFile = filepath.Join(t.TempDir(), "absent", "kept.list")
+	coordinator.sign("node/first", fresh)
+	if changed, err := follower.sync(ctx); err == nil || changed {
+		t.Fatalf("with nowhere to keep the list: changed %v, error %v", changed, err)
+	}
+	if !holds(t, copies, held) || holds(t, copies, fresh) {
+		t.Error("the follower acted on a list it could not keep")
+	}
+}

apps/sisyphusd/replication/list.go

@@ -0,0 +1,86 @@
+package replication
+
+import (
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"os"
+
+	"google.golang.org/protobuf/proto"
+
+	pb "github.com/sisyphus-network/Sisyphus/packages/protocol/sisyphus/v1"
+)
+
+// RestoredOwner is who holds the pins on what a coordinator took back from
+// its followers after losing its store. Who had pinned each blob before was
+// known only to the store that was lost; until when was in the list.
+const RestoredOwner = "restored"
+
+// listHeading begins what is signed of a list, so that a signature of one
+// can never be taken for a signature of anything else the node key signs.
+const listHeading = "sisyphus keep list v1\n"
+
+// signedBytes returns what a coordinator signs of a list: the store, the
+// sequence, and each blob with its expiry, in the encoding blob.proto
+// describes. Every text is preceded by its length, so no two lists encode
+// alike.
+func signedBytes(list *pb.KeepList) []byte {
+	data := []byte(listHeading)
+	data = binary.BigEndian.AppendUint64(data, uint64(len(list.GetStore())))
+	data = append(data, list.GetStore()...)
+	data = binary.BigEndian.AppendUint64(data, list.GetSequence())
+	data = binary.BigEndian.AppendUint64(data, uint64(len(list.GetBlobs())))
+	for _, blob := range list.GetBlobs() {
+		data = binary.BigEndian.AppendUint64(data, uint64(len(blob.GetCid())))
+		data = append(data, blob.GetCid()...)
+		if blob.GetKeepUntil() == nil {
+			data = append(data, 0)
+			continue
+		}
+		data = append(data, 1)
+		data = binary.BigEndian.AppendUint64(data, uint64(blob.GetKeepUntil().AsTime().UnixNano()))
+	}
+	return data
+}
+
+// loadList reads the list a follower keeps in file. A missing file means it
+// has none.
+func loadList(file string) (*pb.KeepList, error) {
+	data, err := os.ReadFile(file)
+	if errors.Is(err, os.ErrNotExist) {
+		return nil, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+	list := new(pb.KeepList)
+	if err := proto.Unmarshal(data, list); err != nil {
+		return nil, fmt.Errorf("%s: %w", file, err)
+	}
+	return list, nil
+}
+
+// saveList puts a list in file in place of the one there.
+func saveList(file string, list *pb.KeepList) error {
+	data, _ := proto.Marshal(list) // a message that was just decoded always encodes
+
+	// Write beside the file and rename, so a crash leaves the old list or
+	// the new one and never half of either.
+	tmp := file + ".tmp"
+	if err := writeSynced(tmp, data); err != nil {
+		return err
+	}
+	return os.Rename(tmp, file)
+}
+
+func writeSynced(path string, data []byte) error {
+	f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
+	if err != nil {
+		return err
+	}
+	_, err = f.Write(data)
+	if err == nil {
+		err = f.Sync()
+	}
+	return errors.Join(err, f.Close())
+}

apps/sisyphusd/replication/list_test.go

@@ -0,0 +1,527 @@
+package replication
+
+import (
+	"os"
+	"path/filepath"
+	"slices"
+	"testing"
+	"time"
+
+	"github.com/ipfs/go-cid"
+	"google.golang.org/protobuf/proto"
+	"google.golang.org/protobuf/types/known/timestamppb"
+
+	"github.com/sisyphus-network/Sisyphus/packages/identity"
+	pb "github.com/sisyphus-network/Sisyphus/packages/protocol/sisyphus/v1"
+	"github.com/sisyphus-network/Sisyphus/packages/storage"
+)
+
+// These tests check that a coordinator signs what it tells followers to
+// hold, and takes back from a list only what it signed for.
+
+// listed is one line of a list: a blob, and until when it is kept. The zero
+// time is no expiry.
+func listed(id string, until time.Time) *pb.KeptBlob {
+	blob := &pb.KeptBlob{Cid: id}
+	if !until.IsZero() {
+		blob.KeepUntil = timestamppb.New(until)
+	}
+	return blob
+}
+
+// signedBy returns a list of a store, signed with a node's key as that node
+// would sign it.
+func signedBy(ident *identity.Identity, store string, blobs ...*pb.KeptBlob) *pb.KeepList {
+	list := &pb.KeepList{Store: store, Sequence: uint64(time.Now().UnixNano()), Blobs: blobs}
+	list.Signature = ident.Sign(signedBytes(list))
+	return list
+}
+
+// show has a follower that keeps lists say what it holds for the store the
+// node had before, and show a list, and returns what it is told.
+func (r *rig) show(id string, list *pb.KeepList, holding ...string) *pb.ReplicateResponse {
+	return r.Replicate(ctx, id, &pb.ReplicateRequest{Store: r.old, Holding: holding, KeepsLists: true, List: list})
+}
+
+// restoredPins lists the pins a store holds on what was taken back, as the
+// expiry of each by CID.
+func restoredPins(store *storage.Store) map[string]time.Time {
+	pins := make(map[string]time.Time)
+	for _, pin := range store.Pins() {
+		if pin.Owner == RestoredOwner {
+			pins[pin.CID.String()] = pin.Expires
+		}
+	}
+	return pins
+}
+
+// cidsOf names the blobs of a list, in order.
+func cidsOf(list *pb.KeepList) []string {
+	var ids []string
+	for _, blob := range list.GetBlobs() {
+		ids = append(ids, blob.GetCid())
+	}
+	slices.Sort(ids)
+	return ids
+}
+
+func TestAFollowerThatKeepsListsIsToldWhatToHoldInOneTheNodeSigned(t *testing.T) {
+	graceOf(t, time.Minute)
+	r := newRig(t, 1)
+	day, week := time.Now().Add(24*time.Hour), time.Now().Add(7*24*time.Hour)
+	open, twice, lasting := put(t, r.store, "kept until released"), put(t, r.store, "kept by two, for a day and for a week"), put(t, r.store, "kept by two, one of them until released")
+	put(t, r.store, "stored, and never pinned")
+	for _, pin := range []storage.Pin{
+		{CID: cid.MustParse(open), Owner: "user"},
+		{CID: cid.MustParse(twice), Owner: "job:1", Expires: week},
+		{CID: cid.MustParse(twice), Owner: "user", Expires: day},
+		{CID: cid.MustParse(lasting), Owner: "job:1", Expires: day},
+		{CID: cid.MustParse(lasting), Owner: "user"},
+	} {
+		if err := r.store.Pin(ctx, pin.Owner, pin.Expires, pin.CID); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	told := r.Replicate(ctx, "a", &pb.ReplicateRequest{KeepsLists: true})
+	list := told.GetList()
+	if len(told.GetHold()) != 0 || told.GetShowList() || told.GetStore() != r.now || list.GetStore() != r.now {
+		t.Fatalf("a follower that keeps lists is told %v, want a list for the node's store and nothing beside it", told)
+	}
+	want := map[string]time.Time{open: {}, twice: week, lasting: {}}
+	if len(list.GetBlobs()) != len(want) {
+		t.Errorf("the list names %v, want the three pinned blobs", cidsOf(list))
+	}
+	for _, blob := range list.GetBlobs() {
+		until, named := want[blob.GetCid()]
+		if !named {
+			t.Errorf("the list names %s, which is not pinned", blob.GetCid())
+		} else if until.IsZero() != (blob.GetKeepUntil() == nil) || (!until.IsZero() && !blob.GetKeepUntil().AsTime().Equal(until)) {
+			t.Errorf("the list keeps %s until %v, want the latest of its pins' expiries, %v", blob.GetCid(), blob.GetKeepUntil(), until)
+		}
+	}
+	// Anyone can check the signature against the node's ID, and the node
+	// knows it for its own.
+	if valid, err := identity.Verify(r.ident.ID(), signedBytes(list), list.GetSignature()); err != nil || !valid {
+		t.Errorf("the list's signature is not the node's: valid %v, error %v", valid, err)
+	}
+	if !r.signedHere(list) {
+		t.Error("the node does not know a list it signed")
+	}
+
+	// Each list is later than the last, however soon it is asked for.
+	last := list.GetSequence()
+	for range 50 {
+		next := r.Replicate(ctx, "a", &pb.ReplicateRequest{KeepsLists: true, Store: r.now, Holding: cidsOf(list)}).GetList().GetSequence()
+		if next <= last {
+			t.Fatalf("a list has sequence %d after one with %d", next, last)
+		}
+		last = next
+	}
+	// A follower that does not say it keeps lists is told as before.
+	if told := r.Replicate(ctx, "b", &pb.ReplicateRequest{}); told.GetList() != nil || len(told.GetHold()) == 0 {
+		t.Errorf("a follower that keeps no lists is told %v", told)
+	}
+}
+
+func TestAListChangedAfterItWasSignedIsNotTheNodes(t *testing.T) {
+	graceOf(t, time.Minute)
+	r := newRig(t, 1)
+	theirs := storage.NewMemory()
+	r.followers["a"] = theirs
+	kept, planted := put(t, theirs, "what the lost store kept"), put(t, theirs, "what a follower would like kept")
+	lapsed := put(t, theirs, "what the lost store kept, until a while ago")
+	soon, past := time.Now().Add(time.Hour), time.Now().Add(-time.Hour)
+	genuine := func() *pb.KeepList {
+		return signedBy(r.ident, r.old, listed(kept, soon), listed(lapsed, past))
+	}
+	if !r.signedHere(genuine()) {
+		t.Fatal("the node does not know a list signed with its key")
+	}
+
+	changes := map[string]func(*pb.KeepList){
+		"a blob added":                 func(l *pb.KeepList) { l.Blobs = append(l.Blobs, listed(planted, time.Time{})) },
+		"a blob put in place of one":   func(l *pb.KeepList) { l.Blobs[0].Cid = planted },
+		"a blob taken off":             func(l *pb.KeepList) { l.Blobs = l.Blobs[:1] },
+		"blobs put in another order":   func(l *pb.KeepList) { slices.Reverse(l.Blobs) },
+		"an expiry put off":            func(l *pb.KeepList) { l.Blobs[1].KeepUntil = timestamppb.New(soon) },
+		"an expiry removed":            func(l *pb.KeepList) { l.Blobs[1].KeepUntil = nil },
+		"an expiry at the epoch added": func(l *pb.KeepList) { l.Blobs[0].KeepUntil = timestamppb.New(time.Unix(0, 0)) },
+		"another store named":          func(l *pb.KeepList) { l.Store = r.ident.ID() + "/other" },
+		"another sequence":             func(l *pb.KeepList) { l.Sequence++ },
+		"a signature cut short":        func(l *pb.KeepList) { l.Signature = l.Signature[:32] },
+		"no signature":                 func(l *pb.KeepList) { l.Signature = nil },
+	}
+	for name, change := range changes {
+		list := genuine()
+		change(list)
+		if r.signedHere(list) {
+			t.Errorf("a list with %s is taken for one the node signed", name)
+		}
+		told := r.show("a", list, kept, planted, lapsed)
+		if told.GetList() != nil || told.GetShowList() || told.GetStore() != r.old || len(told.GetHold()) != 3 {
+			t.Errorf("shown a list with %s the node answers %v, want the follower left holding what it has for the earlier store", name, told)
+		}
+	}
+	if pins := restoredPins(r.store); len(pins) != 0 {
+		t.Errorf("lists that were changed after signing brought back %v", pins)
+	}
+	for _, id := range []string{kept, planted, lapsed} {
+		if holds(t, r.store, id) {
+			t.Errorf("the node fetched %s on the strength of a list that was changed", id)
+		}
+	}
+	if n := r.logs.count("showed a list of what to hold that this node did not sign"); n != 1 {
+		t.Errorf("a follower showing lists the node did not sign was logged %d times, want once", n)
+	}
+	// What it holds is still there for the node's owner to restore.
+	if status := r.Status(""); status.GetFromEarlierStore() != 3 || status.GetUnsignedFromEarlierStore() != 3 || status.GetRestored() != 0 {
+		t.Errorf("the node reports %v, want three blobs held from an earlier store with no list of its own to vouch for them", status)
+	}
+}
+
+func TestAListSignedWithAnotherKeyIsIgnored(t *testing.T) {
+	graceOf(t, time.Minute)
+	r := newRig(t, 1)
+	theirs := storage.NewMemory()
+	r.followers["a"] = theirs
+	planted := put(t, theirs, "listed by somebody who is not the node")
+	forged := signedBy(newIdentity(t), r.old, listed(planted, time.Time{}))
+
+	// Asked without a list, the follower is told to show the one it has.
+	// With one that is not the node's it is no further on.
+	for range 3 {
+		asked := r.Replicate(ctx, "a", &pb.ReplicateRequest{Store: r.old, Holding: []string{planted}, KeepsLists: true})
+		if !asked.GetShowList() || asked.GetList() != nil || asked.GetStore() != r.old || !slices.Equal(asked.GetHold(), []string{planted}) {
+			t.Fatalf("a follower holding a blob for an earlier store is told %v, want to keep it and show its list", asked)
+		}
+		told := r.show("a", forged, planted)
+		if told.GetShowList() || told.GetList() != nil || told.GetStore() != r.old || !slices.Equal(told.GetHold(), []string{planted}) {
+			t.Fatalf("shown a list another node signed, the node answers %v", told)
+		}
+	}
+	if holds(t, r.store, planted) || len(restoredPins(r.store)) != 0 {
+		t.Error("the node took back a blob named by a list another node signed")
+	}
+	if n := r.logs.count("showed a list of what to hold that this node did not sign"); n != 1 {
+		t.Errorf("the forged list was logged %d times, want once", n)
+	}
+	if n := r.logs.count("holds blobs from a store this node had before"); n != 1 {
+		t.Errorf("what the follower is left holding was logged %d times, want once", n)
+	}
+	if status := r.Status(""); status.GetFromEarlierStore() != 1 || status.GetUnsignedFromEarlierStore() != 1 {
+		t.Errorf("the node reports %v", status)
+	}
+
+	// The owner can still have it fetched, on the follower's word.
+	if restored, failed := r.Restore(ctx, "user"); restored != 1 || len(failed) != 0 {
+		t.Errorf("restoring by hand took back %d blobs and failed on %v", restored, failed)
+	}
+	// After which the follower holds for the store the node has.
+	if told := r.Replicate(ctx, "a", &pb.ReplicateRequest{Store: r.old, Holding: []string{planted}, KeepsLists: true}); told.GetStore() != r.now || !slices.Equal(cidsOf(told.GetList()), []string{planted}) {
+		t.Errorf("after the blob was restored by hand the follower is told %v", told)
+	}
+}
+
+func TestACoordinatorTakesBackWhatAListItSignedNames(t *testing.T) {
+	graceOf(t, time.Minute)
+	r := newRig(t, 1)
+	theirs := storage.NewMemory()
+	r.followers["a"] = theirs
+	week := time.Now().Add(7 * 24 * time.Hour)
+	open, timed := put(t, theirs, "kept until released"), put(t, theirs, "kept for a week")
+	lapsed, unlisted := put(t, theirs, "kept until a minute ago"), put(t, theirs, "held for the earlier store, and on no list")
+	const late = "listed, and not on the follower yet"
+	missing := put(t, storage.NewMemory(), late)
+	list := signedBy(r.ident, r.old, listed(open, time.Time{}), listed(timed, week), listed(lapsed, time.Now().Add(-time.Minute)), listed(missing, time.Time{}), listed("not-a-cid", time.Time{}))
+	holding := []string{open, timed, lapsed, unlisted, missing}
+
+	// On the follower's word alone nothing comes back: it is told to keep
+	// what it has and to show its list.
+	asked := r.Replicate(ctx, "a", &pb.ReplicateRequest{Store: r.old, Holding: holding, KeepsLists: true})
+	if !asked.GetShowList() || len(asked.GetHold()) != 5 || len(r.store.Pins()) != 0 {
+		t.Fatalf("before any list is shown the node answers %v and has pins %+v", asked, r.store.Pins())
+	}
+	if r.logs.count("holds blobs from a store this node had before") != 0 {
+		t.Error("what a follower is left holding was logged before it had the chance to show its list")
+	}
+
+	// Shown the list, the node takes back what it names and can be fetched.
+	// What has lapsed the follower drops. What the list does not name is
+	// neither taken back nor dropped: it is left for the node's owner.
+	told := r.show("a", list, holding...)
+	want := []string{open, timed, missing}
+	slices.Sort(want)
+	got := slices.Clone(told.GetHold())
+	slices.Sort(got)
+	if all := slices.Sorted(slices.Values(append(slices.Clone(want), unlisted))); !slices.Equal(got, all) || told.GetStore() != r.old || told.GetList() != nil || told.GetShowList() {
+		t.Errorf("with one listed blob still to come the follower is told %v, want to hold the two taken back, the one to come and the one on no list, for the earlier store still", told)
+	}
+	pins := restoredPins(r.store)
+	if len(pins) != 2 || !pins[open].IsZero() || !pins[timed].Equal(week) {
+		t.Errorf("the node pinned as restored %v, want one blob until released and one for a week", pins)
+	}
+	for _, pin := range r.store.Pins() {
+		if pin.Owner != RestoredOwner && pin.Owner != storage.GraceOwner {
+			t.Errorf("taking back left a pin %+v", pin)
+		}
+	}
+	if holds(t, r.store, lapsed) || holds(t, r.store, unlisted) {
+		t.Error("the node fetched a blob whose expiry had passed, or one its list does not name")
+	}
+	if status := r.Status(""); status.GetRestored() != 2 || status.GetFromEarlierStore() != 2 || status.GetUnsignedFromEarlierStore() != 1 || status.GetRestoring() {
+		t.Errorf("the node reports %v, want two blobs taken back, one to come that a list of its own names, and one that none does", status)
+	}
+	if r.logs.count("took back from a storage follower") != 1 || r.logs.count("could not take back everything") != 1 {
+		t.Error("what was taken back, and that some could not be, was not logged, once each")
+	}
+
+	// Once the follower has the rest, and no longer holds what was on no
+	// list, the next ask finishes the job, and it is given a list for the
+	// store the node has now.
+	put(t, theirs, late)
+	told = r.show("a", list, want...)
+	if told.GetStore() != r.now || !slices.Equal(cidsOf(told.GetList()), want) || len(told.GetHold()) != 0 {
+		t.Errorf("with everything taken back the follower is told %v, want a list of all three for the new store", told)
+	}
+	for _, blob := range told.GetList().GetBlobs() {
+		if blob.GetCid() == timed && !blob.GetKeepUntil().AsTime().Equal(week) {
+			t.Errorf("the new list keeps the blob that had a week left until %v", blob.GetKeepUntil().AsTime())
+		}
+	}
+	if r.logs.count("holds blobs from a store this node had before") != 1 {
+		t.Error("what the follower was left holding after showing its list was not logged, once")
+	}
+
+	// The owner releases one. The old list, shown again, does not bring it
+	// back while the node runs.
+	if err := r.store.Unpin(RestoredOwner, cid.MustParse(open)); err != nil {
+		t.Fatal(err)
+	}
+	// A pin the node took back that has since run out is no longer counted.
+	if err := r.store.Pin(ctx, RestoredOwner, time.Now().Add(-time.Second), cid.MustParse(missing)); err != nil {
+		t.Fatal(err)
+	}
+	r.show("a", list, want...)
+	if pins := restoredPins(r.store); len(pins) != 2 || !pins[timed].Equal(week) {
+		t.Errorf("after one blob was released and the list shown again the node has restored pins %v", pins)
+	}
+	if status := r.Status(""); status.GetRestored() != 1 {
+		t.Errorf("with one pin released and one run out the node counts %d blobs as taken back, want 1", status.GetRestored())
+	}
+
+	// A node started again knows what it took back by its pins: a list that
+	// would keep a blob for less long changes nothing.
+	again := New(r.config(1))
+	shorter := signedBy(r.ident, r.old, listed(timed, time.Now().Add(time.Hour)))
+	again.Replicate(ctx, "a", &pb.ReplicateRequest{Store: r.old, Holding: want, KeepsLists: true, List: shorter})
+	if pins := restoredPins(r.store); !pins[timed].Equal(week) {
+		t.Errorf("a list with an earlier expiry moved a restored pin to %v", pins[timed])
+	}
+}
+
+func TestOfTwoListsThatNameABlobTheLaterExpiryWins(t *testing.T) {
+	graceOf(t, time.Minute)
+	r := newRig(t, 2)
+	r.followers["a"], r.followers["b"] = storage.NewMemory(), storage.NewMemory()
+	texts := []string{"longer on the second list", "until released on the first list", "longer on the first list", "until released on the second list", "on the second list only"}
+	var ids []string
+	for _, text := range texts {
+		ids = append(ids, put(t, r.followers["a"], text))
+		put(t, r.followers["b"], text)
+	}
+	day, week := time.Now().Add(24*time.Hour), time.Now().Add(7*24*time.Hour)
+	first := signedBy(r.ident, r.old, listed(ids[0], day), listed(ids[1], time.Time{}), listed(ids[2], week), listed(ids[3], day))
+	second := signedBy(r.ident, r.old, listed(ids[0], week), listed(ids[1], day), listed(ids[2], day), listed(ids[3], time.Time{}), listed(ids[4], day))
+
+	r.show("a", first, ids[:4]...)
+	r.show("b", second, ids...)
+	want := map[string]time.Time{ids[0]: week, ids[1]: {}, ids[2]: week, ids[3]: {}, ids[4]: day}
+	pins := restoredPins(r.store)
+	if len(pins) != len(want) {
+		t.Fatalf("two lists brought back %d blobs, want all %d", len(pins), len(want))
+	}
+	for i, id := range ids {
+		if !pins[id].Equal(want[id]) {
+			t.Errorf("the blob %q is kept until %v, want %v", texts[i], pins[id], want[id])
+		}
+	}
+	// Both followers then hold for the new store, each with a list of it.
+	for _, id := range []string{"a", "b"} {
+		if told := r.show(id, first, ids...); told.GetStore() != r.now || len(told.GetList().GetBlobs()) != 5 {
+			t.Errorf("follower %s is told %v, want a list of all five for the new store", id, told)
+		}
+	}
+}
+
+func TestAFollowerThatCannotSupplyWhatItsListNamesIsNotTriedForEveryBlob(t *testing.T) {
+	graceOf(t, time.Minute)
+	r := newRig(t, 1)
+	var ids []*pb.KeptBlob
+	var holding []string
+	const texts = 6
+	for i := range texts {
+		id := put(t, storage.NewMemory(), string(rune('a'+i))+" blob the follower says it holds")
+		ids, holding = append(ids, listed(id, time.Time{})), append(holding, id)
+	}
+	list := signedBy(r.ident, r.old, ids...)
+	tried := 0
+	r.fetching = func() { tried++ }
+
+	// The follower is not connected as a worker yet: nothing is tried, and
+	// it is told to keep everything.
+	told := r.show("a", list, holding...)
+	if len(told.GetHold()) != texts || told.GetStore() != r.old || tried != 0 || r.logs.count("not connected as a worker yet") != 1 {
+		t.Errorf("a follower that cannot be reached is told %v, after %d fetches were tried", told, tried)
+	}
+	// Connected, and without any of them, it is tried for a few and the
+	// rest wait for the next time it asks.
+	r.followers["a"] = storage.NewMemory()
+	told = r.show("a", list, holding...)
+	if len(told.GetHold()) != texts || told.GetStore() != r.old {
+		t.Errorf("a follower that has none of what its list names is told %v", told)
+	}
+	if tried != takeBackTries || r.logs.count("could not take back everything") != 2 {
+		t.Errorf("%d fetches were tried and failed, want %d, and one line in the log for the lot", tried, takeBackTries)
+	}
+	if status := r.Status(""); status.GetFromEarlierStore() != texts || status.GetUnsignedFromEarlierStore() != 0 || status.GetRestored() != 0 {
+		t.Errorf("the node reports %v", status)
+	}
+}
+
+func TestANodeSaysWhileItIsTakingAListBack(t *testing.T) {
+	graceOf(t, time.Minute)
+	r := newRig(t, 1)
+	r.followers["a"] = storage.NewMemory()
+	id := put(t, r.followers["a"], "fetched while the node is asked how it is doing")
+	started, carryOn := make(chan struct{}), make(chan struct{})
+	r.fetching = func() {
+		close(started)
+		<-carryOn
+	}
+	done := make(chan *pb.ReplicateResponse)
+	go func() { done <- r.show("a", signedBy(r.ident, r.old, listed(id, time.Time{})), id) }()
+
+	<-started
+	if status := r.Status(""); !status.GetRestoring() || status.GetRestored() != 0 {
+		t.Errorf("part way through taking a list back the node reports %v", status)
+	}
+	close(carryOn)
+	if told := <-done; told.GetStore() != r.now {
+		t.Errorf("the follower is told %v", told)
+	}
+	if status := r.Status(""); status.GetRestoring() || status.GetRestored() != 1 {
+		t.Errorf("with the list taken back the node reports %v", status)
+	}
+}
+
+func TestAListOfTheStoreTheNodeHasBringsNothingBack(t *testing.T) {
+	graceOf(t, time.Minute)
+	r := newRig(t, 1)
+	r.followers["a"] = storage.NewMemory()
+	pinned := keep(t, r.store, "pinned in the store the node has")
+	dropped := put(t, r.followers["a"], "pinned once in this store, and released")
+	// The follower was told to let the blob go, kept the list that said so,
+	// and stopped before it had done it or taken up the store's new name.
+	list := signedBy(r.ident, r.now, listed(pinned, time.Time{}))
+	replayed := signedBy(r.ident, r.now, listed(dropped, time.Time{}))
+	for _, shown := range []*pb.KeepList{list, replayed} {
+		told := r.show("a", shown, pinned, dropped)
+		if told.GetStore() != r.now || !slices.Equal(cidsOf(told.GetList()), []string{pinned}) {
+			t.Errorf("shown a list of the present store the node answers %v, want only what it has pinned", told)
+		}
+	}
+	if holds(t, r.store, dropped) || len(restoredPins(r.store)) != 0 {
+		t.Error("a list of the store the node has now brought a released blob back")
+	}
+}
+
+func TestCopiesHeldForTheNodeUnderALostKeyAreKeptForItsOwnerToRestore(t *testing.T) {
+	graceOf(t, time.Minute)
+	r := newRig(t, 1)
+	lost := newIdentity(t)
+	config := r.config(1)
+	config.Formerly = lost.ID()
+	m := New(config)
+	theirs := storage.NewMemory()
+	r.followers["a"] = theirs
+	id := put(t, theirs, "kept by the node when it had another key")
+	before := lost.ID() + "/old"
+	list := signedBy(lost, before, listed(id, time.Time{}))
+
+	asked := m.Replicate(ctx, "a", &pb.ReplicateRequest{Store: before, Holding: []string{id}, KeepsLists: true})
+	if !asked.GetShowList() || asked.GetStore() != before || !slices.Equal(asked.GetHold(), []string{id}) {
+		t.Fatalf("a follower holding a blob for the node the owner says this one was is told %v", asked)
+	}
+	// The list is signed with a key the node no longer has. It cannot be
+	// told from a forgery, so nothing comes back until the owner says so.
+	told := m.Replicate(ctx, "a", &pb.ReplicateRequest{Store: before, Holding: []string{id}, KeepsLists: true, List: list})
+	if told.GetStore() != before || !slices.Equal(told.GetHold(), []string{id}) || holds(t, r.store, id) {
+		t.Fatalf("shown a list signed with the lost key the node answers %v", told)
+	}
+	if status := m.Status(""); status.GetUnsignedFromEarlierStore() != 1 || status.GetRestored() != 0 {
+		t.Errorf("the node reports %v", status)
+	}
+	if restored, failed := m.Restore(ctx, "user"); restored != 1 || len(failed) != 0 {
+		t.Fatalf("restoring by hand took back %d blobs and failed on %v", restored, failed)
+	}
+	if !slices.ContainsFunc(r.store.Pins(), func(pin storage.Pin) bool {
+		return pin.CID.String() == id && pin.Owner == "user" && pin.Expires.IsZero()
+	}) {
+		t.Errorf("the blob restored by hand is not pinned for the user until released: %+v", r.store.Pins())
+	}
+
+	// A node that was not told it had another key takes the same copies for
+	// another pool's, and has them dropped.
+	if told := r.Replicate(ctx, "a", &pb.ReplicateRequest{Store: before, Holding: []string{"unpinned"}, KeepsLists: true}); told.GetShowList() || len(told.GetList().GetBlobs()) != 1 {
+		t.Errorf("a node with no earlier key is told nothing of one, and answers %v", told)
+	}
+}
+
+func TestAListIsKeptInAFileAndReplacedWhole(t *testing.T) {
+	file := filepath.Join(t.TempDir(), "kept.list")
+	if list, err := loadList(file); list != nil || err != nil {
+		t.Fatalf("reading a list that was never kept: %v, error %v", list, err)
+	}
+	ident := newIdentity(t)
+	first := signedBy(ident, "node/first", listed("a", time.Time{}), listed("b", time.Now()))
+	second := signedBy(ident, "node/first", listed("c", time.Time{}))
+	for _, list := range []*pb.KeepList{first, second} {
+		if err := saveList(file, list); err != nil {
+			t.Fatal(err)
+		}
+		kept, err := loadList(file)
+		if err != nil || !proto.Equal(kept, list) {
+			t.Errorf("the list read back is %v, error %v, want the one kept, %v", kept, err, list)
+		}
+	}
+	// Nothing but the list is left beside it, and only its owner reads it.
+	entries, err := os.ReadDir(filepath.Dir(file))
+	if err != nil || len(entries) != 1 {
+		t.Errorf("keeping a list twice left %d files, error %v", len(entries), err)
+	}
+	if info, err := os.Stat(file); err != nil || info.Mode().Perm()&0o077 != 0 {
+		t.Errorf("the list is kept with mode %v, error %v", info.Mode(), err)
+	}
+
+	// What cannot be read or kept is an error, not an empty list.
+	if err := os.WriteFile(file, []byte("not a list"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	if list, err := loadList(file); list != nil || err == nil {
+		t.Errorf("reading a file that holds no list: %v, error %v", list, err)
+	}
+	if list, err := loadList(filepath.Dir(file)); list != nil || err == nil {
+		t.Errorf("reading a directory as a list: %v, error %v", list, err)
+	}
+	if err := saveList(filepath.Join(t.TempDir(), "absent", "kept.list"), first); err == nil {
+		t.Error("keeping a list in a directory that is not there succeeded")
+	}
+	inTheWay := filepath.Join(t.TempDir(), "kept.list")
+	if err := os.Mkdir(inTheWay, 0o700); err != nil {
+		t.Fatal(err)
+	}
+	if err := saveList(inTheWay, first); err == nil {
+		t.Error("keeping a list where a directory is in the way succeeded")
+	}
+}

apps/sisyphusd/replication/manager.go

@@ -8,6 +8,11 @@
 // lacks, drops what is no longer listed, and says what it holds. It decides
 // nothing.
 //
+// The coordinator signs each list, and the follower keeps the last one. A
+// coordinator that comes back with its key and without its store is shown
+// the list, knows its own signature, and takes back what the list names. A
+// follower's word alone still brings nothing back.
+//
 // The coordinator's side is a Manager and a follower's a Follower.
 package replication
 
@@ -15,6 +20,7 @@ import (
 	"context"
 	"crypto/sha256"
 	"errors"
+	"fmt"
 	"log/slog"
 	"slices"
 	"sort"
@@ -23,8 +29,10 @@ import (
 	"time"
 
 	"github.com/ipfs/go-cid"
+	"google.golang.org/protobuf/types/known/timestamppb"
 
 	"github.com/sisyphus-network/Sisyphus/apps/sisyphusd/blobclient"
+	"github.com/sisyphus-network/Sisyphus/packages/identity"
 	pb "github.com/sisyphus-network/Sisyphus/packages/protocol/sisyphus/v1"
 	"github.com/sisyphus-network/Sisyphus/packages/storage"
 )
@@ -49,13 +57,21 @@ type Store interface {
 }
 
 type Config struct {
-	// ID is the coordinator's node ID.
+	// Identity is the coordinator's key. Its ID names the node, and it signs
-	ID string
+	// what followers are told to hold. A node without one signs nothing and
+	// takes nothing back by itself.
+	Identity *identity.Identity
 	// Store is the coordinator's store, and StoreID a name for it that is
 	// lost if its pins are: a coordinator found with a store of another
 	// name has lost the one its followers hold copies of.
 	Store   Store
 	StoreID string
+	// Formerly is the ID the node had under a key it has since lost, or "".
+	// What followers hold for a store of that node is kept, as it is for an
+	// earlier store of this one, until Restore takes it back. A list signed
+	// with the lost key cannot be told for the node's own and is not acted
+	// on.
+	Formerly string
 	// Replicas is how many followers should hold each pinned blob. Zero
 	// asks none to.
 	Replicas int
@@ -70,9 +86,11 @@ type Config struct {
 // Manager is the coordinator's side of replication. It hears from the
 // followers, tells each what to hold, and knows what each says it holds.
 type Manager struct {
+	identity *identity.Identity
+	id       string
 	store    Store
 	storeID  string
-	earlier  string // what the name of any earlier store of this node begins with
+	earlier  []string // what the name of any earlier store of this node begins with
 	replicas int
 	address  func(nodeID string) string
 	fetch    func(ctx context.Context, from *pb.BlobHolder, c cid.Cid, into blobclient.Putter) error
@@ -82,11 +100,23 @@ type Manager struct {
 	// atStart holds the blobs that were pinned when the manager started,
 	// and settled is when it stops waiting for followers that may already
 	// hold them.
-	atStart map[string]struct{}
+	atStart map[string]time.Time
 	settled time.Time
 
+	// taking is held while a list is being taken back, so that two followers
+	// with the same blob listed do not race for its expiry.
+	taking sync.Mutex
+
 	mu        sync.Mutex
 	followers map[string]*follower
+	// sequence is the sequence of the last list signed.
+	sequence uint64
+	// restored holds what has been taken back from a list, and the expiry
+	// each blob was given. A list shown again brings back nothing that was
+	// taken once and released since, for as long as the node runs.
+	restored map[string]time.Time
+	// restoring counts the lists being taken back at the moment.
+	restoring int
 }
 
 // follower is what a coordinator knows of one follower.
@@ -96,22 +126,41 @@ type follower struct {
 	// held is what it then said it holds.
 	held map[string]struct{}
 	// stranded lists what it holds from an earlier store of this node that
-	// the present store has not pinned.
+	// the present store has not pinned, and unsigned those of them that no
+	// list it showed, signed by this node, names.
 	stranded []string
+	unsigned []string
+	// warned is whether what it is stranded with has been logged, and
+	// refused whether a list it showed was found not to be this node's.
+	warned  bool
+	refused bool
 }
 
 // New returns a manager for a coordinator that has just started.
 func New(cfg Config) *Manager {
 	m := &Manager{
+		identity:  cfg.Identity,
 		store:     cfg.Store,
-		storeID:   cfg.ID + "/" + cfg.StoreID,
-		earlier:   cfg.ID + "/",
 		replicas:  cfg.Replicas,
 		address:   cfg.Address,
 		fetch:     cfg.Fetch,
 		log:       cfg.Log,
 		grace:     Grace,
 		followers: make(map[string]*follower),
+		restored:  make(map[string]time.Time),
+	}
+	if cfg.Identity != nil {
+		m.id = cfg.Identity.ID()
+	}
+	m.storeID, m.earlier = m.id+"/"+cfg.StoreID, []string{m.id + "/"}
+	if cfg.Formerly != "" {
+		m.earlier = append(m.earlier, cfg.Formerly+"/")
+	}
+	// What an earlier run took back is known by the pins it left.
+	for _, pin := range m.store.Pins() {
+		if pin.Owner == RestoredOwner {
+			m.restored[pin.CID.String()] = pin.Expires
+		}
 	}
 	now := time.Now()
 	m.atStart, _ = m.pinned(now)
@@ -124,11 +173,13 @@ func Off(store Store) *Manager {
 	return New(Config{Store: store, Address: func(string) string { return "" }, Log: slog.New(slog.DiscardHandler)})
 }
 
-// pinned returns the blobs the store is keeping as of now, as a set and in
+// pinned returns the blobs the store is keeping as of now, in order, and
-// order. The short pin every new blob gets does not count: a blob is copied
+// until when each is kept: the latest expiry of the pins on it, or the zero
-// once somebody has asked for it to be kept.
+// time if one of them holds until released. The short pin every new blob
-func (m *Manager) pinned(now time.Time) (set map[string]struct{}, order []string) {
+// gets does not count: a blob is copied once somebody has asked for it to be
-	set = make(map[string]struct{})
+// kept.
+func (m *Manager) pinned(now time.Time) (until map[string]time.Time, order []string) {
+	until = make(map[string]time.Time)
 	for _, pin := range m.store.Pins() {
 		if pin.Owner == storage.GraceOwner || (!pin.Expires.IsZero() && pin.Expires.Before(now)) {
 			continue
@@ -140,12 +191,21 @@ func (m *Manager) pinned(now time.Time) (set map[string]struct{}, order []string
 			continue
 		}
 		id := pin.CID.String()
-		if _, listed := set[id]; !listed {
+		kept, listed := until[id]
-			set[id] = struct{}{}
+		if !listed {
 			order = append(order, id)
 		}
+		if !listed || later(pin.Expires, kept) {
+			until[id] = pin.Expires
+		}
 	}
-	return set, order
+	return until, order
+}
+
+// later reports whether a pin expiring at a outlasts one expiring at b. The
+// zero time is no expiry, and outlasts any other.
+func later(a, b time.Time) bool {
+	return !b.IsZero() && (a.IsZero() || a.After(b))
 }
 
 // liveLocked returns the followers heard from within the grace period, in
@@ -185,9 +245,24 @@ func (m *Manager) chosen(blob string, followers []string) []string {
 }
 
 // Replicate takes a follower's word for what it holds, and for which store,
-// and returns what it should hold from now on and the store to name next
+// and tells it what to hold from now on and the store to name next time.
-// time.
+//
-func (m *Manager) Replicate(caller, store string, holding []string) (hold []string, storeID string) {
+// A follower that shows a list this node signed for an earlier store has
+// what the list names taken back first, so that the answer is to hold it for
+// the store the node has now.
+func (m *Manager) Replicate(ctx context.Context, caller string, req *pb.ReplicateRequest) *pb.ReplicateResponse {
+	store, holding, shown := req.GetStore(), req.GetHolding(), req.GetList()
+	vouched := shown != nil && m.signedHere(shown)
+	// named is what a list of an earlier store names, and owed what of that
+	// could not be taken back this time.
+	var named, owed map[string]struct{}
+	if vouched && shown.GetStore() != m.storeID {
+		owed = m.takeBack(ctx, caller, shown)
+		named = make(map[string]struct{}, len(shown.GetBlobs()))
+		for _, blob := range shown.GetBlobs() {
+			named[blob.GetCid()] = struct{}{}
+		}
+	}
 	now := time.Now()
 	pinned, order := m.pinned(now)
 
@@ -205,6 +280,14 @@ func (m *Manager) Replicate(caller, store string, holding []string) (hold []stri
 	m.followers[caller] = f
 	live := m.liveLocked(now)
 
+	// A list that is not this node's is as good as none. A follower that
+	// shows none this time is asked for it again, and is not logged twice.
+	f.refused = (shown != nil && !vouched) || (shown == nil && before.refused)
+	if f.refused && !before.refused {
+		m.log.Warn("a storage follower showed a list of what to hold that this node did not sign; it is ignored, and nothing is taken back on the strength of it", "node", caller, "store", shown.GetStore())
+	}
+
+	var hold []string
 	for _, blob := range order {
 		chosen := m.chosen(blob, live)
 		mine := slices.Contains(chosen, caller)
@@ -224,21 +307,163 @@ func (m *Manager) Replicate(caller, store string, holding []string) (hold []stri
 	// What a follower holds that is not pinned has been released, and it
 	// drops it, unless it holds it for a store this node had before: then
 	// this node has lost its pins, and that copy may be the only one left.
-	if store != m.storeID && strings.HasPrefix(store, m.earlier) {
+	// For what a list this node signed names there is no need to guess. It
-		for _, id := range holding {
+	// has been taken back, and is pinned; or it could not be fetched this
-			if _, kept := pinned[id]; !kept {
+	// time, and is still to be kept; or it has lapsed, or was released after
-				f.stranded = append(f.stranded, id)
+	// it came back, and goes. A follower that shows a list of the store the
-			}
+	// node has now was told by that list what to drop, and stopped before
+	// it had.
+	current := vouched && shown.GetStore() == m.storeID
+	earlier := !current && store != m.storeID && slices.ContainsFunc(m.earlier, func(prefix string) bool {
+		return strings.HasPrefix(store, prefix)
+	})
+	for _, id := range holding {
+		_, kept := pinned[id]
+		_, listed := named[id]
+		_, due := owed[id]
+		switch {
+		case kept, listed && !due, !listed && !earlier:
+		case listed:
+			f.stranded = append(f.stranded, id)
+		default:
+			f.stranded, f.unsigned = append(f.stranded, id), append(f.unsigned, id)
 		}
-		sort.Strings(f.stranded)
 	}
-	if len(f.stranded) == 0 {
+	sort.Strings(f.stranded)
-		return hold, m.storeID
+
+	// A follower that says it keeps lists is told what to hold in one.
+	lists := req.GetKeepsLists() && m.identity != nil
+	if len(f.stranded) > 0 {
+		// It goes on holding for the earlier store, and keeps the list it
+		// has. If it has yet to show that list, it is asked to, and what it
+		// is left with is logged once it has answered.
+		ask := lists && shown == nil
+		f.warned = before.warned
+		if !ask && !f.warned {
+			m.log.Warn("a storage follower holds blobs from a store this node had before and no longer has pinned; it keeps them until they are restored", "node", caller, "blobs", len(f.stranded), "in_no_list_this_node_signed", len(f.unsigned))
+			f.warned = true
+		}
+		return &pb.ReplicateResponse{Hold: append(hold, f.stranded...), Store: store, ShowList: ask}
 	}
-	if len(before.stranded) == 0 {
+	if lists {
-		m.log.Warn("a storage follower holds blobs from a store this node had before and no longer has pinned; it keeps them until they are restored", "node", caller, "blobs", len(f.stranded))
+		return &pb.ReplicateResponse{Store: m.storeID, List: m.signLocked(now, hold, pinned)}
 	}
-	return append(hold, f.stranded...), store
+	return &pb.ReplicateResponse{Hold: hold, Store: m.storeID}
+}
+
+// signLocked returns a signed list of the given blobs, each with the expiry
+// the store has it pinned until.
+func (m *Manager) signLocked(now time.Time, hold []string, until map[string]time.Time) *pb.KeepList {
+	m.sequence = max(uint64(now.UnixNano()), m.sequence+1)
+	list := &pb.KeepList{Store: m.storeID, Sequence: m.sequence}
+	for _, id := range hold {
+		blob := &pb.KeptBlob{Cid: id}
+		if expires := until[id]; !expires.IsZero() {
+			blob.KeepUntil = timestamppb.New(expires)
+		}
+		list.Blobs = append(list.Blobs, blob)
+	}
+	list.Signature = m.identity.Sign(signedBytes(list))
+	return list
+}
+
+// signedHere reports whether a list carries this node's signature.
+func (m *Manager) signedHere(list *pb.KeepList) bool {
+	if m.identity == nil {
+		return false
+	}
+	signed, _ := identity.Verify(m.id, signedBytes(list), list.GetSignature()) // fails only for an ID that is not one
+	return signed
+}
+
+// takeBackTries is how many blobs in a row may fail to come back before the
+// rest of a list is left for the next time its follower asks. A follower
+// that cannot be reached costs seconds for each blob tried.
+const takeBackTries = 3
+
+// takeBack fetches from a follower what a list signed by this node names,
+// and pins each blob as restored until the expiry the list gives. What has
+// lapsed is passed over. It returns what is still to come: the blobs it
+// could not fetch or pin this time.
+func (m *Manager) takeBack(ctx context.Context, caller string, list *pb.KeepList) (owed map[string]struct{}) {
+	m.taking.Lock()
+	defer m.taking.Unlock()
+	m.mu.Lock()
+	m.restoring++
+	m.mu.Unlock()
+	defer func() {
+		m.mu.Lock()
+		m.restoring--
+		m.mu.Unlock()
+	}()
+
+	now := time.Now()
+	from := &pb.BlobHolder{NodeId: caller, Address: m.address(caller)}
+	owed = make(map[string]struct{})
+	took, failures := 0, 0
+	// first is the first thing to go wrong, which is as much as is logged:
+	// a follower that cannot be reached fails alike for every blob.
+	var first error
+	if from.GetAddress() == "" {
+		failures, first = takeBackTries, errors.New("the follower is not connected as a worker yet, or serves nothing")
+	}
+	for _, blob := range list.GetBlobs() {
+		id := blob.GetCid()
+		var expires time.Time
+		if blob.GetKeepUntil() != nil {
+			expires = blob.GetKeepUntil().AsTime()
+		}
+		if !expires.IsZero() && !expires.After(now) {
+			continue
+		}
+		// Of two lists that name a blob, the one that keeps it longer wins.
+		m.mu.Lock()
+		had, done := m.restored[id]
+		m.mu.Unlock()
+		if done && !later(expires, had) {
+			continue
+		}
+		if failures == takeBackTries {
+			owed[id] = struct{}{}
+			continue
+		}
+		if err := m.takeOne(ctx, from, id, expires); err != nil {
+			if first == nil {
+				first = fmt.Errorf("%s: %w", id, err)
+			}
+			owed[id] = struct{}{}
+			failures++
+			continue
+		}
+		failures = 0
+		took++
+		m.mu.Lock()
+		m.restored[id] = expires
+		m.mu.Unlock()
+	}
+	if len(owed) > 0 {
+		m.log.Warn("could not take back everything a list signed by this node names; the rest is tried again when the follower next asks", "node", caller, "blobs", len(owed), "error", first)
+	}
+	if took > 0 {
+		m.log.Info("took back from a storage follower what a store this node had before was keeping, on the strength of a list this node signed",
+			"node", caller, "store", list.GetStore(), "signed", time.Unix(0, int64(list.GetSequence())).UTC().Format(time.RFC3339), "blobs", took, "still_to_come", len(owed))
+	}
+	return owed
+}
+
+// takeOne pins a blob as restored, fetching it from a follower first if the
+// store lacks it. The store checks what arrives against the CID.
+func (m *Manager) takeOne(ctx context.Context, from *pb.BlobHolder, id string, expires time.Time) error {
+	c, err := cid.Decode(id)
+	if err == nil {
+		err = m.store.Pin(ctx, RestoredOwner, expires, c)
+	}
+	if errors.Is(err, storage.ErrNotFound) {
+		if err = m.fetch(ctx, from, c, m.store); err == nil {
+			err = m.store.Pin(ctx, RestoredOwner, expires, c)
+		}
+	}
+	return err
 }
 
 // allHoldLocked reports whether every one of some followers says it holds a
@@ -265,7 +490,14 @@ func (m *Manager) Status(only string) *pb.ReplicasResponse {
 		Wanted: uint32(m.replicas), Followers: live,
 		Settling: len(m.atStart) > 0 && now.Before(m.settled),
 	}
-	res.FromEarlierStore = uint32(len(m.strandedLocked(live)))
+	res.FromEarlierStore = uint32(len(m.strandedLocked(live, false)))
+	res.UnsignedFromEarlierStore = uint32(len(m.strandedLocked(live, true)))
+	res.Restoring = m.restoring > 0
+	for _, pin := range m.store.Pins() {
+		if pin.Owner == RestoredOwner && (pin.Expires.IsZero() || pin.Expires.After(now)) {
+			res.Restored++
+		}
+	}
 	for _, blob := range order {
 		if only != "" && blob != only {
 			continue
@@ -282,11 +514,15 @@ func (m *Manager) Status(only string) *pb.ReplicasResponse {
 }
 
 // strandedLocked lists, in order, what the given followers hold from an
-// earlier store.
+// earlier store, or only what no list signed by this node names.
-func (m *Manager) strandedLocked(followers []string) []string {
+func (m *Manager) strandedLocked(followers []string, unsigned bool) []string {
 	var stranded []string
 	for _, id := range followers {
-		stranded = append(stranded, m.followers[id].stranded...)
+		if unsigned {
+			stranded = append(stranded, m.followers[id].unsigned...)
+		} else {
+			stranded = append(stranded, m.followers[id].stranded...)
+		}
 	}
 	sort.Strings(stranded)
 	return slices.Compact(stranded)
@@ -325,10 +561,12 @@ func (m *Manager) Recover(ctx context.Context, c cid.Cid) bool {
 
 // Restore fetches back what followers hold from an earlier store of this
 // node and pins it for owner, until released. It returns how many blobs it
-// restored and which it could not.
+// restored and which it could not. It takes the followers' word for what
+// that store kept, so it is for the node's owner to ask for: what a list
+// signed by this node names is taken back without it.
 func (m *Manager) Restore(ctx context.Context, owner string) (restored int, failed []string) {
 	m.mu.Lock()
-	stranded := m.strandedLocked(m.liveLocked(time.Now()))
+	stranded := m.strandedLocked(m.liveLocked(time.Now()), false)
 	m.mu.Unlock()
 
 	done := make(map[string]struct{})

apps/sisyphusd/replication/manager_test.go

@@ -6,6 +6,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"path/filepath"
 	"slices"
 	"strings"
 	"sync"
@@ -15,6 +16,7 @@ import (
 	"github.com/ipfs/go-cid"
 
 	"github.com/sisyphus-network/Sisyphus/apps/sisyphusd/blobclient"
+	"github.com/sisyphus-network/Sisyphus/packages/identity"
 	pb "github.com/sisyphus-network/Sisyphus/packages/protocol/sisyphus/v1"
 	"github.com/sisyphus-network/Sisyphus/packages/storage"
 )
@@ -69,6 +71,16 @@ func keep(t *testing.T, store *storage.Store, text string) string {
 	return id
 }
 
+// newIdentity makes a node key that lasts for the test.
+func newIdentity(t *testing.T) *identity.Identity {
+	t.Helper()
+	ident, _, err := identity.LoadOrCreate(filepath.Join(t.TempDir(), "node.key"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	return ident
+}
+
 // rig is a manager over a store of its own, and the stores of followers it
 // can fetch from by name.
 type rig struct {
@@ -77,18 +89,31 @@ type rig struct {
 	logs  *logged
 	// followers are the stores of the followers that serve blobs, by ID.
 	followers map[string]*storage.Store
+	// ident is the node's key, now the name of the store it has, and old
+	// that of one it had before.
+	ident    *identity.Identity
+	now, old string
+	// fetching, if set, is called as each fetch from a follower begins.
+	fetching func()
 }
 
 // newRig returns a manager, wanting so many copies, of a store that already
 // has each of the given texts pinned.
 func newRig(t *testing.T, replicas int, pinned ...string) *rig {
 	t.Helper()
-	r := &rig{store: storage.NewMemory(), logs: new(logged), followers: make(map[string]*storage.Store)}
+	r := &rig{store: storage.NewMemory(), logs: new(logged), followers: make(map[string]*storage.Store), ident: newIdentity(t)}
+	r.now, r.old = r.ident.ID()+"/new", r.ident.ID()+"/old"
 	for _, text := range pinned {
 		keep(t, r.store, text)
 	}
-	r.Manager = New(Config{
+	r.Manager = New(r.config(replicas))
-		ID: "node", Store: r.store, StoreID: "new", Replicas: replicas, Log: r.logs.logger(),
+	return r
+}
+
+// config returns what the rig's manager is made with.
+func (r *rig) config(replicas int) Config {
+	return Config{
+		Identity: r.ident, Store: r.store, StoreID: "new", Replicas: replicas, Log: r.logs.logger(),
 		Address: func(id string) string {
 			if r.followers[id] == nil {
 				return ""
@@ -96,6 +121,9 @@ func newRig(t *testing.T, replicas int, pinned ...string) *rig {
 			return id + ":7701"
 		},
 		Fetch: func(ctx context.Context, from *pb.BlobHolder, c cid.Cid, into blobclient.Putter) error {
+			if r.fetching != nil {
+				r.fetching()
+			}
 			blob, err := r.followers[from.GetNodeId()].Open(ctx, c)
 			if err != nil {
 				return err
@@ -104,14 +132,20 @@ func newRig(t *testing.T, replicas int, pinned ...string) *rig {
 			_, err = into.Put(ctx, blob)
 			return err
 		},
-	})
+	}
-	return r
+}
+
+// ask has a follower that keeps no lists say what it holds, and for which
+// store, and returns what it is told to hold and for which.
+func (r *rig) ask(id, store string, holding []string) (hold []string, name string) {
+	told := r.Replicate(ctx, id, &pb.ReplicateRequest{Store: store, Holding: holding})
+	return told.GetHold(), told.GetStore()
 }
 
 // obey has a follower ask what to hold, saying it holds what it was last
 // told to, and returns what it is told now.
 func (r *rig) obey(id string, holding []string) []string {
-	hold, _ := r.Replicate(id, "node/new", holding)
+	hold, _ := r.ask(id, r.now, holding)
 	return hold
 }
 
@@ -368,12 +402,12 @@ func TestCopiesOfAStoreThatWasLostAreKeptUntilRestored(t *testing.T) {
 	holding := []string{servable, unservable, pinned, "not-a-cid"}
 	slices.Sort(holding)
 
-	hold, store := r.Replicate("a", "node/old", holding)
+	hold, store := r.ask("a", r.old, holding)
 	slices.Sort(hold)
-	if !slices.Equal(hold, holding) || store != "node/old" {
+	if !slices.Equal(hold, holding) || store != r.old {
 		t.Fatalf("a follower with copies of a lost store is to hold %v for %q, want everything it has, for the old store still", hold, store)
 	}
-	r.Replicate("a", "node/old", holding)
+	r.ask("a", r.old, holding)
 	if r.logs.count("holds blobs from a store this node had before") != 1 {
 		t.Error("finding copies of a lost store was not logged, once")
 	}
@@ -399,7 +433,7 @@ func TestCopiesOfAStoreThatWasLostAreKeptUntilRestored(t *testing.T) {
 	}
 	// While anything is left, the follower goes on holding for the old
 	// store.
-	if _, store := r.Replicate("a", "node/old", holding); store != "node/old" {
+	if _, store := r.ask("a", r.old, holding); store != r.old {
 		t.Errorf("with blobs still to restore the follower is told its store is %q", store)
 	}
 
@@ -407,12 +441,12 @@ func TestCopiesOfAStoreThatWasLostAreKeptUntilRestored(t *testing.T) {
 	// and the follower moves on to the new store.
 	put(t, theirs, "held by the follower, it says")
 	holding = slices.DeleteFunc(holding, func(id string) bool { return id == "not-a-cid" })
-	r.Replicate("a", "node/old", holding)
+	r.ask("a", r.old, holding)
 	if restored, failed := r.Restore(ctx, "user"); restored != 1 || len(failed) != 0 {
 		t.Errorf("restoring again took back %d blobs and failed on %v, want 1 and none", restored, failed)
 	}
-	hold, store = r.Replicate("a", "node/old", holding)
+	hold, store = r.ask("a", r.old, holding)
-	if store != "node/new" || !slices.Equal(hold, holding) {
+	if store != r.now || !slices.Equal(hold, holding) {
 		t.Errorf("after everything was restored the follower is to hold %v for %q, want all three for the new store", hold, store)
 	}
 	if status := r.Status(""); status.GetFromEarlierStore() != 0 || len(status.GetBlobs()) != 3 {
@@ -425,9 +459,9 @@ func TestCopiesOfAnotherNodesStoreAreNotKept(t *testing.T) {
 	r := newRig(t, 1)
 	pinned := keep(t, r.store, "pinned")
 	foreign := put(t, storage.NewMemory(), "held for a pool this follower was in before")
-	for _, store := range []string{"other-node/old", "", "node/new"} {
+	for _, store := range []string{"other-node/old", "", r.now} {
-		hold, now := r.Replicate("a", store, []string{foreign})
+		hold, now := r.ask("a", store, []string{foreign})
-		if !slices.Equal(hold, []string{pinned}) || now != "node/new" {
+		if !slices.Equal(hold, []string{pinned}) || now != r.now {
 			t.Errorf("a follower holding a blob for store %q is to hold %v for %q, want only what this node has pinned", store, hold, now)
 		}
 	}
@@ -452,7 +486,7 @@ func TestALostBlobIsFetchedBackFromAFollowerThatCanSupplyIt(t *testing.T) {
 	// and the third does.
 	r.followers["b"], r.followers["c"] = storage.NewMemory(), storage.NewMemory()
 	for _, id := range []string{"a", "b", "c"} {
-		r.Replicate(id, "node/new", []string{lost})
+		r.ask(id, r.now, []string{lost})
 	}
 	if _, err := kept.Open(ctx, c); !errors.Is(err, storage.ErrNotFound) {
 		t.Fatalf("opening a blob no follower can supply: %v", err)
@@ -486,9 +520,11 @@ func TestANodeThatAsksNothingOfFollowersGivesThemNothingToHold(t *testing.T) {
 	store := storage.NewMemory()
 	pinned := keep(t, store, "pinned on a node that keeps no copies")
 	off := Off(store)
-	hold, name := off.Replicate("a", "", []string{pinned})
+	// It has no key to sign with, so even a follower that keeps lists is
-	if len(hold) != 0 || name != "/" {
+	// given none, and a list shown to it counts for nothing.
-		t.Errorf("a follower of a node that keeps no copies is to hold %v for %q", hold, name)
+	told := off.Replicate(ctx, "a", &pb.ReplicateRequest{Holding: []string{pinned}, KeepsLists: true, List: &pb.KeepList{Store: "/earlier"}})
+	if len(told.GetHold()) != 0 || told.GetStore() != "/" || told.GetList() != nil || told.GetShowList() {
+		t.Errorf("a follower of a node that keeps no copies is told %v", told)
 	}
 	if holders := off.Holders(pinned, ""); len(holders) != 0 {
 		t.Errorf("the blob's holders are said to be %v", holders)

apps/sisyphusd/replication_test.go

@@ -392,7 +392,7 @@ func TestACoordinatorFetchesBackABlobItsStoreHasLost(t *testing.T) {
 
 // These run real nodes, started as the command line starts them.
 
-func TestACoordinatorThatLostItsStoreRestoresItFromAFollower(t *testing.T) {
+func TestACoordinatorThatLostItsStoreTakesItBackFromAFollower(t *testing.T) {
 	replicateQuickly(t, time.Minute)
 	addr, coordinatorDir, copies := freeAddr(t), t.TempDir(), filepath.Join(t.TempDir(), "copies")
 	stop := startDaemon(t, "--listen", addr, "--data-dir", coordinatorDir, "--replicas", "1", "--slots", "1")
@@ -400,12 +400,24 @@ func TestACoordinatorThatLostItsStoreRestoresItFromAFollower(t *testing.T) {
 	startDaemon(t, "--role", "worker", "--coordinator", addr, "--replica-dir", copies, "--name", "follower", "--slots", "1")
 	waitForOutput(t, "follower", "nodes", "--addr", addr)
 
-	const text = "the only thing this pool was ever asked to keep"
+	const text, passing = "the one thing this pool was asked to keep for good", "and the one it was asked to keep for three days"
 	id := strings.TrimSpace(mustCLI(t, "blob", "put", "--addr", addr, writeFile(t, text)))
-	waitForOutput(t, "  1/1  ", "blob", "replicas", "--addr", addr)
+	timed := strings.TrimSpace(mustCLI(t, "blob", "put", "--addr", addr, "--ttl", "72h", writeFile(t, passing)))
-	if out := mustCLI(t, "blob", "replicas", "--addr", addr); strings.Contains(out, "too few") || strings.Contains(out, "fewer copies") {
+	waitFor(t, func() bool { return strings.Count(mustCLI(t, "blob", "replicas", "--addr", addr), "  1/1  ") == 2 })
+	if out := mustCLI(t, "blob", "replicas", "--addr", addr); strings.Contains(out, "too few") || strings.Contains(out, "fewer copies") || strings.Contains(out, "took back") {
 		t.Errorf("with one copy wanted and one held, blob replicas says:\n%s", out)
 	}
+	// The pin on the second, as "blob pins" prints it: for the user, until
+	// a time three days off.
+	until := slices.DeleteFunc(pinLines(t, addr, timed), func(line string) bool { return !strings.HasPrefix(line, "user 2") })
+	if len(until) != 1 {
+		t.Fatalf("the blob stored for three days has pins %v", pinLines(t, addr, timed))
+	}
+	// The follower has on its disk, beside the copies, what the coordinator
+	// signed.
+	if _, err := os.Stat(filepath.Join(copies, "kept.list")); err != nil {
+		t.Errorf("the follower keeps no signed list: %v", err)
+	}
 
 	// The coordinator comes back with the same key and none of its stored
 	// data: the disk it was on has been replaced.
@@ -415,38 +427,98 @@ func TestACoordinatorThatLostItsStoreRestoresItFromAFollower(t *testing.T) {
 	}
 	startDaemon(t, "--listen", addr, "--data-dir", coordinatorDir, "--replicas", "1", "--slots", "1")
 
-	// The follower is not told to drop what the lost store had pinned, and
+	// Nobody runs anything. Once the follower has found its way to the
-	// the coordinator says what it finds.
+	// restarted node again, which is looked to every ten seconds, the node
-	out := waitForOutput(t, "from a store this node had before", "blob", "replicas", "--addr", addr)
+	// takes back what it had signed for, each blob until when it was kept.
-	if !strings.Contains(out, "followers hold 1 blob(s)") || !strings.Contains(out, "nothing is pinned") {
+	waitFor(t, func() bool {
-		t.Errorf("after losing its store the coordinator reports:\n%s", out)
+		out, err := cli(t, "blob", "pins", "--addr", addr)
+		return err == nil && strings.Count(out, "  restored  ") == 2
+	})
+	if !has(pinLines(t, addr, id), "restored released") || !has(pinLines(t, addr, timed), strings.Replace(until[0], "user", "restored", 1)) {
+		t.Errorf("what came back has pins %v and %v, want one until released and one until when it was kept before, %v", pinLines(t, addr, id), pinLines(t, addr, timed), until)
+	}
+	for blob, want := range map[string]string{id: text, timed: passing} {
+		if out := mustCLI(t, "blob", "get", "--addr", addr, blob); out != want {
+			t.Errorf("the blob taken back reads %q, want %q", out, want)
+		}
+		if lines := pinLines(t, addr, blob); slices.ContainsFunc(lines, func(line string) bool { return strings.HasPrefix(line, "user") }) {
+			t.Errorf("the blob taken back has pins %v, want none for the user: who had pinned it was not in the list", lines)
+		}
+	}
+	// The follower then holds them for the new store, as it did for the old.
+	var out string
+	waitFor(t, func() bool {
+		out = mustCLI(t, "blob", "replicas", "--addr", addr)
+		return strings.Count(out, "  1/1  ") == 2 && !strings.Contains(out, "from a store this node had before")
+	})
+	if !strings.Contains(out, `2 blob(s) are kept that this node took back from its followers after losing its store`) {
+		t.Errorf("after taking its data back the coordinator reports:\n%s", out)
+	}
+	// There is nothing left for the command to do, and it says why.
+	out = mustCLI(t, "blob", "restore", "--addr", addr)
+	if !strings.HasPrefix(out, "restored 0 blob(s), pinned until unpinned\nfollowers hold nothing from an earlier store") || !strings.Contains(out, "takes back by itself what it had signed for") {
+		t.Errorf("blob restore with nothing left to restore printed %q", out)
+	}
+
+	// What came back is the owner's to let go of.
+	mustCLI(t, "blob", "unpin", "--addr", addr, id)
+	if lines := pinLines(t, addr, id); has(lines, "restored released") {
+		t.Errorf("after blob unpin the blob taken back still has pins %v", lines)
+	}
+	waitFor(t, func() bool {
+		out := mustCLI(t, "blob", "replicas", "--addr", addr)
+		return strings.Count(out, "  1/1  ") == 1 && strings.Contains(out, "1 blob(s) are kept that this node took back")
+	})
+}
+
+func TestACoordinatorWithANewKeyRestoresOnItsOwnersWordWhatAFollowerHoldsForItsOldOne(t *testing.T) {
+	replicateQuickly(t, time.Minute)
+	addr, lostDir, followerDir, copies := freeAddr(t), t.TempDir(), t.TempDir(), filepath.Join(t.TempDir(), "copies")
+	stop := startDaemon(t, "--listen", addr, "--data-dir", lostDir, "--replicas", "1", "--slots", "1")
+	stopFollower := startDaemon(t, "--role", "worker", "--coordinator", addr, "--data-dir", followerDir, "--replica-dir", copies, "--name", "follower", "--slots", "1")
+	waitForOutput(t, "follower", "nodes", "--addr", addr)
+	const text = "kept by a coordinator whose whole disk was then lost"
+	id := strings.TrimSpace(mustCLI(t, "blob", "put", "--addr", addr, "--ttl", "72h", writeFile(t, text)))
+	waitForOutput(t, "  1/1  ", "blob", "replicas", "--addr", addr)
+	was, err := loadIdentity(lostDir)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// The coordinator's disk is gone, key and all. Its owner starts another
+	// in its place, tells it the ID the first had, and has the follower join
+	// it: to the follower it is another node at the address of the one it
+	// knew.
+	stopFollower()
+	stop()
+	startDaemon(t, "--listen", addr, "--replicas", "1", "--slots", "1", "--formerly", was.ID())
+	mustCLI(t, "pool", "join", "--addr", addr, "--data-dir", followerDir, invite(t, addr, "worker"))
+	startDaemon(t, "--role", "worker", "--coordinator", addr, "--data-dir", followerDir, "--replica-dir", copies, "--name", "follower", "--slots", "1")
+
+	// The follower's list is signed with a key the new node does not have,
+	// so nothing comes back by itself, and the node says what would.
+	out := waitForOutput(t, "that no list signed by this node names", "blob", "replicas", "--addr", addr)
+	if !strings.Contains(out, "followers hold 1 blob(s)") || !strings.Contains(out, `"sisyphusd blob restore" does`) || !strings.Contains(out, "nothing is pinned") {
+		t.Errorf("a coordinator with a new key reports:\n%s", out)
 	}
 	if out := mustCLI(t, "blob", "pins", "--addr", addr); !strings.Contains(out, "nothing is pinned") {
-		t.Fatalf("the coordinator's new store already has pins:\n%s", out)
+		t.Fatalf("the new coordinator already has pins:\n%s", out)
 	}
-	// Asked for the blob, it fetches it back, once the follower has found
-	// its way to the restarted node again: that is looked to every ten
-	// seconds.
-	waitFor(t, func() bool {
-		out, err := cli(t, "blob", "get", "--addr", addr, id)
-		return err == nil && out == text
-	})
 
-	// And on its owner's word it takes back everything the follower holds.
+	// On its owner's word it takes back everything the follower holds, for
-	if out := mustCLI(t, "blob", "restore", "--addr", addr); out != "restored 1 blob(s), pinned until unpinned\n" {
+	// the user and until unpinned: the three days are not believed.
-		t.Errorf("blob restore printed %q", out)
+	waitForOutput(t, "restored 1 blob(s), pinned until unpinned\n", "blob", "restore", "--addr", addr)
-	}
 	if lines := pinLines(t, addr, id); !has(lines, "user released") {
 		t.Errorf("the restored blob has pins %v, want one held for the user until released", lines)
 	}
-	// The follower then holds it for the new store, as it did for the old.
+	if out := mustCLI(t, "blob", "get", "--addr", addr, id); out != text {
+		t.Errorf("the restored blob reads %q", out)
+	}
+	// The follower then holds it for the new node.
 	waitFor(t, func() bool {
 		out := mustCLI(t, "blob", "replicas", "--addr", addr)
 		return strings.Contains(out, "  1/1  ") && !strings.Contains(out, "from a store this node had before")
 	})
-	if out := mustCLI(t, "blob", "restore", "--addr", addr); out != "restored 0 blob(s), pinned until unpinned\n" {
-		t.Errorf("blob restore with nothing left to restore printed %q", out)
-	}
 }
 
 func TestReplicationFlagsAreChecked(t *testing.T) {
@@ -474,6 +546,8 @@ func TestReplicationFlagsAreChecked(t *testing.T) {
 		{[]string{"--replicas", "-1"}, "--replicas is a number of copies"},
 		{[]string{"--coordinator", addr, "--role", "worker", "--replicas", "2"}, "a worker-only node offers to hold copies with --replica-dir"},
 		{[]string{"--replica-dir", t.TempDir()}, "--replica-dir is for worker-only nodes"},
+		{[]string{"--formerly", "not-a-node-id"}, `--formerly: node ID "not-a-node-id"`},
+		{[]string{"--coordinator", addr, "--role", "worker", "--formerly", newIdentity(t).ID()}, "--formerly is for a node that coordinates a pool"},
 		{[]string{"--coordinator", addr, "--role", "worker", "--join", invite(t, addr, "worker"), "--replica-dir", writeFile(t, "not a directory")}, "--replica-dir: open blob store"},
 		{[]string{"--listen", freeAddr(t), "--data-dir", nameIsDir}, "read the store's name"},
 		{[]string{"--listen", freeAddr(t), "--data-dir", nameCannotBeSaved}, "save the store's name"},
@@ -515,9 +589,21 @@ func TestBlobReplicasPrintsWhatTheNodeReports(t *testing.T) {
 	}{
 		{"a node that keeps no copies", &pb.ReplicasResponse{Blobs: []*pb.ReplicatedBlob{{Cid: helloCID}}},
 			"this node asks no followers to hold copies of its data: it was started without --replicas\n"},
-		{"one that keeps none, and whose followers hold what it lost", &pb.ReplicasResponse{FromEarlierStore: 2},
+		{"one that keeps none, and whose followers hold what it lost and signed for nothing of", &pb.ReplicasResponse{FromEarlierStore: 2, UnsignedFromEarlierStore: 2},
-			"followers hold 2 blob(s) from a store this node had before, which it no longer has pinned: \"sisyphusd blob restore\" fetches them back\n" +
+			"followers hold 2 blob(s) from a store this node had before that no list signed by this node names: it does not take those back by itself, and \"sisyphusd blob restore\" does\n" +
 				"this node asks no followers to hold copies of its data: it was started without --replicas\n"},
+		{"one that is taking back what it lost", &pb.ReplicasResponse{Wanted: 1, Followers: []string{"a"}, Restoring: true, Restored: 4, FromEarlierStore: 7, UnsignedFromEarlierStore: 7},
+			"this node is taking back what a follower holds from a store it had before: 4 blob(s) are back so far\n" +
+				"followers hold 7 blob(s) from a store this node had before that no list signed by this node names: it does not take those back by itself, and \"sisyphusd blob restore\" does\n" +
+				"1 follower(s) connected; each pinned blob is to be held by 1\n" +
+				"nothing is pinned\n"},
+		{"one that has taken most of it back, and has some to come and some it will not take", &pb.ReplicasResponse{Wanted: 1, Followers: []string{"a"}, Restored: 4, FromEarlierStore: 3, UnsignedFromEarlierStore: 1, Blobs: []*pb.ReplicatedBlob{{Cid: "back", Holders: []string{"a"}}}},
+			"4 blob(s) are kept that this node took back from its followers after losing its store: \"sisyphusd blob pins\" lists them as held for \"restored\"\n" +
+				"followers hold 2 blob(s) from a store this node had before that it has yet to take back: it tries again each time a follower asks what to hold, and there is nothing to run\n" +
+				"followers hold 1 blob(s) from a store this node had before that no list signed by this node names: it does not take those back by itself, and \"sisyphusd blob restore\" does\n" +
+				"1 follower(s) connected; each pinned blob is to be held by 1\n" +
+				"CID   COPIES  HELD BY\n" +
+				"back  1/1     a\n"},
 		{"one that has just started, with nothing pinned", &pb.ReplicasResponse{Wanted: 2, Followers: []string{"a", "b"}, Settling: true},
 			"2 follower(s) connected; each pinned blob is to be held by 2\n" +
 				"the node started a short while ago: followers that hold copies from before are given time to say so before any are handed what it already had\n" +

apps/sisyphusd/restore_test.go

@@ -0,0 +1,273 @@
+package main
+
+import (
+	"os"
+	"slices"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/ipfs/go-cid"
+	"google.golang.org/protobuf/proto"
+
+	"github.com/sisyphus-network/Sisyphus/apps/sisyphusd/coordinator"
+	"github.com/sisyphus-network/Sisyphus/apps/sisyphusd/replication"
+	pb "github.com/sisyphus-network/Sisyphus/packages/protocol/sisyphus/v1"
+	"github.com/sisyphus-network/Sisyphus/packages/runtime"
+	"github.com/sisyphus-network/Sisyphus/packages/storage"
+)
+
+// These tests check that a coordinator which has lost its store takes back
+// from its followers what it had signed for, and nothing else.
+
+// comesBack stops a pool's coordinator and returns the pool of the one that
+// takes its place: it has an empty store under another name, the key and
+// whatever else as says, and the first one's followers, each with the copies
+// and the signed list it had.
+func (p *pool) comesBack(as replication.Config) *pool {
+	p.t.Helper()
+	for _, unfollow := range p.unfollow {
+		unfollow()
+	}
+	p.stop()
+	as.StoreID = "second"
+	q := startPoolAs(p.t, runtime.Builtin(), func(store *storage.Store) coordinator.Store { return store }, nil, as)
+	for name, copies := range p.copies {
+		q.startMember(name, 1, copies, p.lists[name])
+	}
+	q.waitForWorkers(len(p.copies))
+	return q
+}
+
+// keptUntil returns until when a store keeps each blob it has pinned for an
+// owner, by CID. The zero time is until released.
+func keptUntil(store *storage.Store, owner string) map[string]time.Time {
+	until := make(map[string]time.Time)
+	for _, pin := range store.Pins() {
+		if pin.Owner == owner {
+			until[pin.CID.String()] = pin.Expires
+		}
+	}
+	return until
+}
+
+// keptList reads the list a follower keeps.
+func (p *pool) keptList(name string) *pb.KeepList {
+	p.t.Helper()
+	data, err := os.ReadFile(p.lists[name])
+	list := new(pb.KeepList)
+	if err == nil {
+		err = proto.Unmarshal(data, list)
+	}
+	if err != nil {
+		p.t.Fatal(err)
+	}
+	return list
+}
+
+func TestACoordinatorThatLostItsStoreTakesBackWhatItSignedForWithoutBeingAsked(t *testing.T) {
+	replicateQuickly(t, time.Minute)
+	p := startReplicatedPool(t, 2, "a", "b", "c")
+	texts := map[string]string{}
+	keep := func(text string, pins ...storage.Pin) string {
+		t.Helper()
+		id := p.upload(text)
+		for _, pin := range pins {
+			if err := p.store.Pin(p.ctx, pin.Owner, pin.Expires, cid.MustParse(id)); err != nil {
+				t.Fatal(err)
+			}
+		}
+		texts[id] = text
+		return id
+	}
+	day, week := time.Now().Add(24*time.Hour), time.Now().Add(7*24*time.Hour)
+	open := keep("kept until released", storage.Pin{Owner: "user"})
+	timed := keep("kept for a day", storage.Pin{Owner: "user", Expires: day})
+	twice := keep("kept by a job for a week and by its owner for a day", storage.Pin{Owner: "job:1", Expires: week}, storage.Pin{Owner: "user", Expires: day})
+	released := keep("released before the store was lost", storage.Pin{Owner: "user"})
+	p.waitForCopies(2)
+	if _, err := p.blobs.Unpin(p.ctx, &pb.UnpinBlobRequest{Cid: released}); err != nil {
+		t.Fatal(err)
+	}
+	waitFor(t, func() bool { return len(p.copiesOf(released)) == 0 })
+	// One is kept for a short while only, and the followers stop asking
+	// before it has run out: they still hold it, and their lists name it.
+	lapses := time.Now().Add(3 * time.Second)
+	brief := keep("kept for a moment", storage.Pin{Owner: "job:2", Expires: lapses})
+	waitFor(t, func() bool { return len(p.copiesOf(brief)) == 2 || time.Now().After(lapses) })
+
+	// The store is lost. The coordinator that comes back has the same key.
+	q := p.comesBack(replication.Config{Identity: p.ident, Replicas: 2})
+	time.Sleep(time.Until(lapses))
+	want := map[string]time.Time{open: {}, timed: day, twice: week}
+	waitFor(t, func() bool {
+		got := keptUntil(q.store, replication.RestoredOwner)
+		for id := range want {
+			if _, back := got[id]; !back {
+				return false
+			}
+		}
+		return true
+	})
+	// Each list names two thirds of what was kept, and between them they
+	// name everything. What two lists name comes back once.
+	got := keptUntil(q.store, replication.RestoredOwner)
+	for id, until := range want {
+		if !got[id].Equal(until) {
+			t.Errorf("%q is kept until %v, want %v, as the store that was lost had it", texts[id], got[id], until)
+		}
+		if read := stored(t, q.store, cid.MustParse(id)); string(read) != texts[id] {
+			t.Errorf("%q came back as %q", texts[id], read)
+		}
+	}
+	for _, pin := range q.store.Pins() {
+		if pin.Owner != replication.RestoredOwner && pin.Owner != storage.GraceOwner {
+			t.Errorf("the store has a pin %+v, want everything that came back held as restored", pin)
+		}
+		if id := pin.CID.String(); id == released || (id == brief && pin.Owner == replication.RestoredOwner && pin.Expires.After(lapses)) {
+			t.Errorf("the store has a pin %+v on %q", pin, texts[id])
+		}
+	}
+	if q.holds(released) {
+		t.Error("a blob released before the store was lost came back")
+	}
+
+	// The followers carry on for the new store, and drop what had lapsed.
+	q.waitForCopies(2)
+	waitFor(t, func() bool {
+		listed := q.replicas()
+		return len(q.copiesOf(brief)) == 0 && listed.GetFromEarlierStore() == 0 && !listed.GetRestoring()
+	})
+	for name, store := range q.copies {
+		for _, pin := range store.Pins() {
+			if pin.Owner != storage.GraceOwner && pin.Owner != "pool:"+p.ident.ID()+"/second" {
+				t.Errorf("follower %s has a pin %+v, want every one held for the coordinator's new store", name, pin)
+			}
+		}
+		if list := q.keptList(name); list.GetStore() != p.ident.ID()+"/second" {
+			t.Errorf("follower %s keeps a list for %q, want one for the new store", name, list.GetStore())
+		}
+	}
+	if listed := q.replicas(); listed.GetRestored() != 3 || len(listed.GetBlobs()) != 3 || listed.GetUnsignedFromEarlierStore() != 0 {
+		t.Errorf("the coordinator reports %v, want three blobs taken back and copied", listed)
+	}
+	if !strings.Contains(q.logs.String(), "took back from a storage follower") || strings.Contains(q.logs.String(), "did not sign") {
+		t.Errorf("the coordinator logged:\n%s", q.logs.String())
+	}
+
+	// Its owner sees what happened, has nothing to run, and can let go of
+	// what came back.
+	client := q.clientOf()
+	if out := mustCLI(t, "blob", "replicas", "--addr", q.addr, "--data-dir", client); !strings.Contains(out, "3 blob(s) are kept that this node took back from its followers after losing its store") || strings.Contains(out, "blob restore") {
+		t.Errorf("blob replicas says:\n%s", out)
+	}
+	if out := mustCLI(t, "blob", "restore", "--addr", q.addr, "--data-dir", client); !strings.HasPrefix(out, "restored 0 blob(s), pinned until unpinned\nfollowers hold nothing from an earlier store") {
+		t.Errorf("blob restore printed %q", out)
+	}
+	mustCLI(t, "blob", "unpin", "--addr", q.addr, "--data-dir", client, open)
+	waitFor(t, func() bool { return len(q.copiesOf(open)) == 0 })
+	if _, kept := keptUntil(q.store, replication.RestoredOwner)[open]; kept {
+		t.Error("blob unpin left the pin on a blob that was taken back")
+	}
+}
+
+func TestAFollowerThatAddsToItsListBringsNothingBack(t *testing.T) {
+	replicateQuickly(t, time.Minute)
+	p := startReplicatedPool(t, 1, "a")
+	kept := p.keep("what the coordinator chose to keep")
+	p.waitForCopies(1)
+	p.unfollow["a"]()
+
+	// The follower puts a blob of its own among the copies, and on the list
+	// the coordinator signed.
+	planted, err := p.copies["a"].Put(p.ctx, strings.NewReader("what a follower would like the pool to keep"))
+	if err == nil {
+		err = p.copies["a"].Pin(p.ctx, "pool:"+p.ident.ID()+"/first", time.Time{}, planted)
+	}
+	list := p.keptList("a")
+	if len(list.GetBlobs()) != 1 || len(list.GetSignature()) == 0 {
+		t.Fatalf("the follower keeps the list %v, want the one blob, signed", list)
+	}
+	list.Blobs = append(list.Blobs, &pb.KeptBlob{Cid: planted.String()})
+	data, _ := proto.Marshal(list)
+	if err == nil {
+		err = os.WriteFile(p.lists["a"], data, 0o600)
+	}
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	q := p.comesBack(replication.Config{Identity: p.ident, Replicas: 1})
+	waitFor(t, func() bool {
+		return strings.Contains(q.logs.String(), "showed a list of what to hold that this node did not sign")
+	})
+	// Nothing comes back, not even what the list named before it was added
+	// to: the list is the node's or it is not.
+	waitFor(t, func() bool { return q.replicas().GetUnsignedFromEarlierStore() == 2 })
+	for range 20 {
+		if pins := q.store.Pins(); len(pins) != 0 || q.holds(kept) || q.holds(planted.String()) {
+			t.Fatalf("a list that was added to brought back pins %+v", pins)
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	if strings.Contains(q.logs.String(), "took back") {
+		t.Error("the coordinator logged that it took something back")
+	}
+	// The follower is left holding both, for the owner to decide.
+	if holders := q.copiesOf(kept); !slices.Equal(holders, []string{"a"}) {
+		t.Errorf("the blob the coordinator had kept is now held by %v", holders)
+	}
+	out := mustCLI(t, "blob", "replicas", "--addr", q.addr, "--data-dir", q.clientOf())
+	if !strings.Contains(out, `followers hold 2 blob(s) from a store this node had before that no list signed by this node names`) || !strings.Contains(out, `"sisyphusd blob restore" does`) || strings.Contains(out, "took back") {
+		t.Errorf("blob replicas says:\n%s", out)
+	}
+}
+
+func TestACoordinatorThatLostItsKeyTooRestoresOnlyOnItsOwnersWord(t *testing.T) {
+	replicateQuickly(t, time.Minute)
+	p := startReplicatedPool(t, 1, "a")
+	const text = "kept by a coordinator that then lost its disk, key and all"
+	id := p.upload(text)
+	if _, err := p.blobs.Pin(p.ctx, &pb.PinBlobRequest{Cid: id, TtlSeconds: 3600}); err != nil {
+		t.Fatal(err)
+	}
+	p.waitForCopies(1)
+
+	// The coordinator that comes back is another node, as far as anyone can
+	// check. Its owner has told it the ID it had, so it keeps what the
+	// follower holds for that node, and takes none of it back.
+	q := p.comesBack(replication.Config{Identity: newIdentity(t), Formerly: p.ident.ID(), Replicas: 1})
+	waitFor(t, func() bool {
+		return strings.Contains(q.logs.String(), "showed a list of what to hold that this node did not sign")
+	})
+	waitFor(t, func() bool { return q.replicas().GetUnsignedFromEarlierStore() == 1 })
+	if pins := q.store.Pins(); len(pins) != 0 || q.holds(id) {
+		t.Fatalf("a coordinator with a new key took back pins %+v on the strength of a list signed with its old one", pins)
+	}
+	if holders := q.copiesOf(id); !slices.Equal(holders, []string{"a"}) {
+		t.Fatalf("the blob is now held by %v, want the follower to have kept it", holders)
+	}
+	client := q.clientOf()
+	if out := mustCLI(t, "blob", "replicas", "--addr", q.addr, "--data-dir", client); !strings.Contains(out, `followers hold 1 blob(s) from a store this node had before that no list signed by this node names`) {
+		t.Errorf("blob replicas says:\n%s", out)
+	}
+
+	// The command still does what it did: everything back, for the user,
+	// until unpinned. Until when the lost store kept it is not believed.
+	if out := mustCLI(t, "blob", "restore", "--addr", q.addr, "--data-dir", client); out != "restored 1 blob(s), pinned until unpinned\n" {
+		t.Errorf("blob restore printed %q", out)
+	}
+	if got := stored(t, q.store, cid.MustParse(id)); string(got) != text {
+		t.Errorf("the blob came back as %q", got)
+	}
+	if until, pinned := keptUntil(q.store, "user")[id]; !pinned || !until.IsZero() || len(keptUntil(q.store, replication.RestoredOwner)) != 0 {
+		t.Errorf("the restored blob has pins %+v, want one for the user until released", q.store.Pins())
+	}
+	// The follower then holds it for the new node, with a list that node
+	// signed.
+	q.waitForCopies(1)
+	waitFor(t, func() bool { return q.keptList("a").GetStore() == q.ident.ID()+"/second" })
+	if listed := q.replicas(); listed.GetFromEarlierStore() != 0 || listed.GetRestored() != 0 {
+		t.Errorf("the coordinator reports %v", listed)
+	}
+}

docs/development.md

@@ -49,7 +49,7 @@ CI runs formatting, `go vet`, the tests under the race detector, `make cover` an
 | `apps/sisyphusd/mcpserver` | The node as a Model Context Protocol server: ten tools over the local API, for agents other than the node's own. Tested against a real node and against one that fails. |
 | `apps/sisyphusd/planner` | The planner's loop: the model is asked, the tools it asks for are run, and it is asked again with what they returned. Tested against a scripted model. |
 | `apps/sisyphusd/blobclient` | Uploading and downloading blobs, checked against their CIDs. |
-| `apps/sisyphusd/replication` | Copies of a pool's stored data on its storage followers. The coordinator's side says which followers hold each pinned blob, by rendezvous hashing, and fetches a blob back; the follower's side holds what it is told. Tested without a network, and in `apps/sisyphusd` with real nodes. |
+| `apps/sisyphusd/replication` | Copies of a pool's stored data on its storage followers. The coordinator's side says which followers hold each pinned blob, by rendezvous hashing, signs what it tells each to hold, fetches a blob back, and takes back what a list it signed names when it has lost its store; the follower's side holds what it is told and keeps the last signed list. Tested without a network, and in `apps/sisyphusd` with real nodes. |
 | `apps/sisyphusd/pinning` | The IPFS Pinning Service API, both ways: the service a coordinator offers with `--pinning-listen`, and the client behind `blob pin-remote`. The service is tested against a store and a Kubo that can be made to fail; the real `ipfs pin remote` commands are run against a node in `apps/sisyphusd/pinning_test.go`. |
 | `apps/sisyphusd` | The `sisyphusd` command: flags, startup, the command-line client. |
 | `apps/sisyphus` | The desktop app: Electron and React. `npm ci`, then `npm run typecheck` and `npm run build`. |

packages/protocol/sisyphus/v1/blob.pb.go

@@ -689,7 +689,8 @@ type Pin struct {
 	Cid   string                 `protobuf:"bytes,1,opt,name=cid,proto3" json:"cid,omitempty"`
 	// Who holds the pin: "user", "job:<id>", or "recent" for the short pin
 	// every newly stored blob gets. In a follower's store of copies it is
-	// "pool:" and the ID of the store they are copies of.
+	// "pool:" and the ID of the store they are copies of. "restored" is on
+	// what a node that lost its store took back from its followers.
 	Owner string `protobuf:"bytes,2,opt,name=owner,proto3" json:"owner,omitempty"`
 	// Unset for a pin that holds until released.
 	ExpiresAt     *timestamppb.Timestamp `protobuf:"bytes,3,opt,name=expires_at,json=expiresAt,proto3" json:"expires_at,omitempty"`
@@ -850,7 +851,13 @@ type ReplicateRequest struct {
 	Holding []string `protobuf:"bytes,1,rep,name=holding,proto3" json:"holding,omitempty"`
 	// The store those are copies of, as the node named it in its last answer.
 	// Empty for a follower that holds nothing.
-	Store         string `protobuf:"bytes,2,opt,name=store,proto3" json:"store,omitempty"`
+	Store string `protobuf:"bytes,2,opt,name=store,proto3" json:"store,omitempty"`
+	// The last list the node signed for this follower, handed back when the
+	// node asked to be shown it.
+	List *KeepList `protobuf:"bytes,3,opt,name=list,proto3" json:"list,omitempty"`
+	// Set by a follower that keeps the node's signed lists. It is then told
+	// what to hold in a list alone, and hold is left empty.
+	KeepsLists    bool `protobuf:"varint,4,opt,name=keeps_lists,json=keepsLists,proto3" json:"keeps_lists,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
@@ -899,12 +906,36 @@ func (x *ReplicateRequest) GetStore() string {
 	return ""
 }
 
+func (x *ReplicateRequest) GetList() *KeepList {
+	if x != nil {
+		return x.List
+	}
+	return nil
+}
+
+func (x *ReplicateRequest) GetKeepsLists() bool {
+	if x != nil {
+		return x.KeepsLists
+	}
+	return false
+}
+
 type ReplicateResponse struct {
 	state protoimpl.MessageState `protogen:"open.v1"`
 	// Everything the follower should hold from now on, and nothing else.
+	// Empty if list is set: the list then says the same.
 	Hold []string `protobuf:"bytes,1,rep,name=hold,proto3" json:"hold,omitempty"`
 	// What to give as the store next time.
-	Store         string `protobuf:"bytes,2,opt,name=store,proto3" json:"store,omitempty"`
+	Store string `protobuf:"bytes,2,opt,name=store,proto3" json:"store,omitempty"`
+	// What the follower should hold, signed, for it to keep in place of the
+	// list it had. Unset while the follower holds blobs from an earlier store
+	// that the node has not taken back: the list it has is then the only
+	// proof of what that store kept.
+	List *KeepList `protobuf:"bytes,3,opt,name=list,proto3" json:"list,omitempty"`
+	// Set when the follower holds blobs from a store other than the node's
+	// present one and gave no list. It should ask again at once, with the
+	// list it has.
+	ShowList      bool `protobuf:"varint,4,opt,name=show_list,json=showList,proto3" json:"show_list,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
@@ -953,6 +984,153 @@ func (x *ReplicateResponse) GetStore() string {
 	return ""
 }
 
+func (x *ReplicateResponse) GetList() *KeepList {
+	if x != nil {
+		return x.List
+	}
+	return nil
+}
+
+func (x *ReplicateResponse) GetShowList() bool {
+	if x != nil {
+		return x.ShowList
+	}
+	return false
+}
+
+// KeepList is a node's statement of what one follower is to hold, signed so
+// that the node can later tell it for its own.
+type KeepList struct {
+	state protoimpl.MessageState `protogen:"open.v1"`
+	// The store the list is of: the node's ID, "/", and the store's name.
+	Store string `protobuf:"bytes,1,opt,name=store,proto3" json:"store,omitempty"`
+	// When the node signed the list, in nanoseconds since 1970, or one more
+	// than its last if its clock had not moved on. It only goes up.
+	Sequence uint64      `protobuf:"varint,2,opt,name=sequence,proto3" json:"sequence,omitempty"`
+	Blobs    []*KeptBlob `protobuf:"bytes,3,rep,name=blobs,proto3" json:"blobs,omitempty"`
+	// The node's signature, with its node key, of this encoding of the fields
+	// above, in which every number is eight bytes, most significant first:
+	// the text "sisyphus keep list v1" and a newline; the length of store and
+	// store; the sequence; the number of blobs; and for each blob the length
+	// of its CID, the CID, and either a zero byte, if it has no keep_until,
+	// or a one byte and keep_until in nanoseconds since 1970.
+	Signature     []byte `protobuf:"bytes,4,opt,name=signature,proto3" json:"signature,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *KeepList) Reset() {
+	*x = KeepList{}
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[20]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *KeepList) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*KeepList) ProtoMessage() {}
+
+func (x *KeepList) ProtoReflect() protoreflect.Message {
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[20]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use KeepList.ProtoReflect.Descriptor instead.
+func (*KeepList) Descriptor() ([]byte, []int) {
+	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{20}
+}
+
+func (x *KeepList) GetStore() string {
+	if x != nil {
+		return x.Store
+	}
+	return ""
+}
+
+func (x *KeepList) GetSequence() uint64 {
+	if x != nil {
+		return x.Sequence
+	}
+	return 0
+}
+
+func (x *KeepList) GetBlobs() []*KeptBlob {
+	if x != nil {
+		return x.Blobs
+	}
+	return nil
+}
+
+func (x *KeepList) GetSignature() []byte {
+	if x != nil {
+		return x.Signature
+	}
+	return nil
+}
+
+type KeptBlob struct {
+	state protoimpl.MessageState `protogen:"open.v1"`
+	Cid   string                 `protobuf:"bytes,1,opt,name=cid,proto3" json:"cid,omitempty"`
+	// The latest expiry of the node's pins on the blob. Unset if one of them
+	// holds until released.
+	KeepUntil     *timestamppb.Timestamp `protobuf:"bytes,2,opt,name=keep_until,json=keepUntil,proto3" json:"keep_until,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *KeptBlob) Reset() {
+	*x = KeptBlob{}
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[21]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *KeptBlob) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*KeptBlob) ProtoMessage() {}
+
+func (x *KeptBlob) ProtoReflect() protoreflect.Message {
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[21]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use KeptBlob.ProtoReflect.Descriptor instead.
+func (*KeptBlob) Descriptor() ([]byte, []int) {
+	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{21}
+}
+
+func (x *KeptBlob) GetCid() string {
+	if x != nil {
+		return x.Cid
+	}
+	return ""
+}
+
+func (x *KeptBlob) GetKeepUntil() *timestamppb.Timestamp {
+	if x != nil {
+		return x.KeepUntil
+	}
+	return nil
+}
+
 type ReplicasRequest struct {
 	state protoimpl.MessageState `protogen:"open.v1"`
 	// If set, report on this blob only.
@@ -963,7 +1141,7 @@ type ReplicasRequest struct {
 
 func (x *ReplicasRequest) Reset() {
 	*x = ReplicasRequest{}
-	mi := &file_sisyphus_v1_blob_proto_msgTypes[20]
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[22]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -975,7 +1153,7 @@ func (x *ReplicasRequest) String() string {
 func (*ReplicasRequest) ProtoMessage() {}
 
 func (x *ReplicasRequest) ProtoReflect() protoreflect.Message {
-	mi := &file_sisyphus_v1_blob_proto_msgTypes[20]
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[22]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -988,7 +1166,7 @@ func (x *ReplicasRequest) ProtoReflect() protoreflect.Message {
 
 // Deprecated: Use ReplicasRequest.ProtoReflect.Descriptor instead.
 func (*ReplicasRequest) Descriptor() ([]byte, []int) {
-	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{20}
+	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{22}
 }
 
 func (x *ReplicasRequest) GetCid() string {
@@ -1010,15 +1188,23 @@ type ReplicasResponse struct {
 	// before are given time to say so first.
 	Settling bool `protobuf:"varint,4,opt,name=settling,proto3" json:"settling,omitempty"`
 	// How many blobs followers hold from a store the node had before and
-	// has not pinned in the one it has now. Restore takes them back.
+	// has not pinned in the one it has now.
 	FromEarlierStore uint32 `protobuf:"varint,5,opt,name=from_earlier_store,json=fromEarlierStore,proto3" json:"from_earlier_store,omitempty"`
-	unknownFields    protoimpl.UnknownFields
+	// Set while the node is taking back what a list it signed names.
-	sizeCache        protoimpl.SizeCache
+	Restoring bool `protobuf:"varint,6,opt,name=restoring,proto3" json:"restoring,omitempty"`
+	// How many blobs the node has pinned as "restored": those it took back
+	// by itself and is still keeping.
+	Restored uint32 `protobuf:"varint,7,opt,name=restored,proto3" json:"restored,omitempty"`
+	// How many of from_earlier_store no list signed by the node covers. The
+	// node does not take these back by itself; Restore does.
+	UnsignedFromEarlierStore uint32 `protobuf:"varint,8,opt,name=unsigned_from_earlier_store,json=unsignedFromEarlierStore,proto3" json:"unsigned_from_earlier_store,omitempty"`
+	unknownFields            protoimpl.UnknownFields
+	sizeCache                protoimpl.SizeCache
 }
 
 func (x *ReplicasResponse) Reset() {
 	*x = ReplicasResponse{}
-	mi := &file_sisyphus_v1_blob_proto_msgTypes[21]
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[23]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -1030,7 +1216,7 @@ func (x *ReplicasResponse) String() string {
 func (*ReplicasResponse) ProtoMessage() {}
 
 func (x *ReplicasResponse) ProtoReflect() protoreflect.Message {
-	mi := &file_sisyphus_v1_blob_proto_msgTypes[21]
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[23]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -1043,7 +1229,7 @@ func (x *ReplicasResponse) ProtoReflect() protoreflect.Message {
 
 // Deprecated: Use ReplicasResponse.ProtoReflect.Descriptor instead.
 func (*ReplicasResponse) Descriptor() ([]byte, []int) {
-	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{21}
+	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{23}
 }
 
 func (x *ReplicasResponse) GetWanted() uint32 {
@@ -1081,6 +1267,27 @@ func (x *ReplicasResponse) GetFromEarlierStore() uint32 {
 	return 0
 }
 
+func (x *ReplicasResponse) GetRestoring() bool {
+	if x != nil {
+		return x.Restoring
+	}
+	return false
+}
+
+func (x *ReplicasResponse) GetRestored() uint32 {
+	if x != nil {
+		return x.Restored
+	}
+	return 0
+}
+
+func (x *ReplicasResponse) GetUnsignedFromEarlierStore() uint32 {
+	if x != nil {
+		return x.UnsignedFromEarlierStore
+	}
+	return 0
+}
+
 type ReplicatedBlob struct {
 	state protoimpl.MessageState `protogen:"open.v1"`
 	Cid   string                 `protobuf:"bytes,1,opt,name=cid,proto3" json:"cid,omitempty"`
@@ -1092,7 +1299,7 @@ type ReplicatedBlob struct {
 
 func (x *ReplicatedBlob) Reset() {
 	*x = ReplicatedBlob{}
-	mi := &file_sisyphus_v1_blob_proto_msgTypes[22]
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[24]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -1104,7 +1311,7 @@ func (x *ReplicatedBlob) String() string {
 func (*ReplicatedBlob) ProtoMessage() {}
 
 func (x *ReplicatedBlob) ProtoReflect() protoreflect.Message {
-	mi := &file_sisyphus_v1_blob_proto_msgTypes[22]
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[24]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -1117,7 +1324,7 @@ func (x *ReplicatedBlob) ProtoReflect() protoreflect.Message {
 
 // Deprecated: Use ReplicatedBlob.ProtoReflect.Descriptor instead.
 func (*ReplicatedBlob) Descriptor() ([]byte, []int) {
-	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{22}
+	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{24}
 }
 
 func (x *ReplicatedBlob) GetCid() string {
@@ -1142,7 +1349,7 @@ type RestoreRequest struct {
 
 func (x *RestoreRequest) Reset() {
 	*x = RestoreRequest{}
-	mi := &file_sisyphus_v1_blob_proto_msgTypes[23]
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[25]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -1154,7 +1361,7 @@ func (x *RestoreRequest) String() string {
 func (*RestoreRequest) ProtoMessage() {}
 
 func (x *RestoreRequest) ProtoReflect() protoreflect.Message {
-	mi := &file_sisyphus_v1_blob_proto_msgTypes[23]
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[25]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -1167,7 +1374,7 @@ func (x *RestoreRequest) ProtoReflect() protoreflect.Message {
 
 // Deprecated: Use RestoreRequest.ProtoReflect.Descriptor instead.
 func (*RestoreRequest) Descriptor() ([]byte, []int) {
-	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{23}
+	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{25}
 }
 
 type RestoreResponse struct {
@@ -1182,7 +1389,7 @@ type RestoreResponse struct {
 
 func (x *RestoreResponse) Reset() {
 	*x = RestoreResponse{}
-	mi := &file_sisyphus_v1_blob_proto_msgTypes[24]
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[26]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -1194,7 +1401,7 @@ func (x *RestoreResponse) String() string {
 func (*RestoreResponse) ProtoMessage() {}
 
 func (x *RestoreResponse) ProtoReflect() protoreflect.Message {
-	mi := &file_sisyphus_v1_blob_proto_msgTypes[24]
+	mi := &file_sisyphus_v1_blob_proto_msgTypes[26]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -1207,7 +1414,7 @@ func (x *RestoreResponse) ProtoReflect() protoreflect.Message {
 
 // Deprecated: Use RestoreResponse.ProtoReflect.Descriptor instead.
 func (*RestoreResponse) Descriptor() ([]byte, []int) {
-	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{24}
+	return file_sisyphus_v1_blob_proto_rawDescGZIP(), []int{26}
 }
 
 func (x *RestoreResponse) GetRestored() uint32 {
@@ -1271,21 +1478,38 @@ const file_sisyphus_v1_blob_proto_rawDesc = "" +
 	"\fexpired_pins\x18\x01 \x01(\rR\vexpiredPins\x12%\n" +
 	"\x0eblocks_removed\x18\x02 \x01(\x04R\rblocksRemoved\x12\x1f\n" +
 	"\vbytes_freed\x18\x03 \x01(\x04R\n" +
-	"bytesFreed\"B\n" +
+	"bytesFreed\"\x8e\x01\n" +
 	"\x10ReplicateRequest\x12\x18\n" +
 	"\aholding\x18\x01 \x03(\tR\aholding\x12\x14\n" +
-	"\x05store\x18\x02 \x01(\tR\x05store\"=\n" +
+	"\x05store\x18\x02 \x01(\tR\x05store\x12)\n" +
+	"\x04list\x18\x03 \x01(\v2\x15.sisyphus.v1.KeepListR\x04list\x12\x1f\n" +
+	"\vkeeps_lists\x18\x04 \x01(\bR\n" +
+	"keepsLists\"\x85\x01\n" +
 	"\x11ReplicateResponse\x12\x12\n" +
 	"\x04hold\x18\x01 \x03(\tR\x04hold\x12\x14\n" +
-	"\x05store\x18\x02 \x01(\tR\x05store\"#\n" +
+	"\x05store\x18\x02 \x01(\tR\x05store\x12)\n" +
+	"\x04list\x18\x03 \x01(\v2\x15.sisyphus.v1.KeepListR\x04list\x12\x1b\n" +
+	"\tshow_list\x18\x04 \x01(\bR\bshowList\"\x87\x01\n" +
+	"\bKeepList\x12\x14\n" +
+	"\x05store\x18\x01 \x01(\tR\x05store\x12\x1a\n" +
+	"\bsequence\x18\x02 \x01(\x04R\bsequence\x12+\n" +
+	"\x05blobs\x18\x03 \x03(\v2\x15.sisyphus.v1.KeptBlobR\x05blobs\x12\x1c\n" +
+	"\tsignature\x18\x04 \x01(\fR\tsignature\"W\n" +
+	"\bKeptBlob\x12\x10\n" +
+	"\x03cid\x18\x01 \x01(\tR\x03cid\x129\n" +
+	"\n" +
+	"keep_until\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\tkeepUntil\"#\n" +
 	"\x0fReplicasRequest\x12\x10\n" +
-	"\x03cid\x18\x01 \x01(\tR\x03cid\"\xc5\x01\n" +
+	"\x03cid\x18\x01 \x01(\tR\x03cid\"\xbe\x02\n" +
 	"\x10ReplicasResponse\x12\x16\n" +
 	"\x06wanted\x18\x01 \x01(\rR\x06wanted\x12\x1c\n" +
 	"\tfollowers\x18\x02 \x03(\tR\tfollowers\x121\n" +
 	"\x05blobs\x18\x03 \x03(\v2\x1b.sisyphus.v1.ReplicatedBlobR\x05blobs\x12\x1a\n" +
 	"\bsettling\x18\x04 \x01(\bR\bsettling\x12,\n" +
-	"\x12from_earlier_store\x18\x05 \x01(\rR\x10fromEarlierStore\"<\n" +
+	"\x12from_earlier_store\x18\x05 \x01(\rR\x10fromEarlierStore\x12\x1c\n" +
+	"\trestoring\x18\x06 \x01(\bR\trestoring\x12\x1a\n" +
+	"\brestored\x18\a \x01(\rR\brestored\x12=\n" +
+	"\x1bunsigned_from_earlier_store\x18\b \x01(\rR\x18unsignedFromEarlierStore\"<\n" +
 	"\x0eReplicatedBlob\x12\x10\n" +
 	"\x03cid\x18\x01 \x01(\tR\x03cid\x12\x18\n" +
 	"\aholders\x18\x02 \x03(\tR\aholders\"\x10\n" +
@@ -1318,7 +1542,7 @@ func file_sisyphus_v1_blob_proto_rawDescGZIP() []byte {
 	return file_sisyphus_v1_blob_proto_rawDescData
 }
 
-var file_sisyphus_v1_blob_proto_msgTypes = make([]protoimpl.MessageInfo, 25)
+var file_sisyphus_v1_blob_proto_msgTypes = make([]protoimpl.MessageInfo, 27)
 var file_sisyphus_v1_blob_proto_goTypes = []any{
 	(*PutBlobRequest)(nil),         // 0: sisyphus.v1.PutBlobRequest
 	(*PutBlobResponse)(nil),        // 1: sisyphus.v1.PutBlobResponse
@@ -1340,45 +1564,51 @@ var file_sisyphus_v1_blob_proto_goTypes = []any{
 	(*CollectGarbageResponse)(nil), // 17: sisyphus.v1.CollectGarbageResponse
 	(*ReplicateRequest)(nil),       // 18: sisyphus.v1.ReplicateRequest
 	(*ReplicateResponse)(nil),      // 19: sisyphus.v1.ReplicateResponse
-	(*ReplicasRequest)(nil),        // 20: sisyphus.v1.ReplicasRequest
+	(*KeepList)(nil),               // 20: sisyphus.v1.KeepList
-	(*ReplicasResponse)(nil),       // 21: sisyphus.v1.ReplicasResponse
+	(*KeptBlob)(nil),               // 21: sisyphus.v1.KeptBlob
-	(*ReplicatedBlob)(nil),         // 22: sisyphus.v1.ReplicatedBlob
+	(*ReplicasRequest)(nil),        // 22: sisyphus.v1.ReplicasRequest
-	(*RestoreRequest)(nil),         // 23: sisyphus.v1.RestoreRequest
+	(*ReplicasResponse)(nil),       // 23: sisyphus.v1.ReplicasResponse
-	(*RestoreResponse)(nil),        // 24: sisyphus.v1.RestoreResponse
+	(*ReplicatedBlob)(nil),         // 24: sisyphus.v1.ReplicatedBlob
-	(*timestamppb.Timestamp)(nil),  // 25: google.protobuf.Timestamp
+	(*RestoreRequest)(nil),         // 25: sisyphus.v1.RestoreRequest
+	(*RestoreResponse)(nil),        // 26: sisyphus.v1.RestoreResponse
+	(*timestamppb.Timestamp)(nil),  // 27: google.protobuf.Timestamp
 }
 var file_sisyphus_v1_blob_proto_depIdxs = []int32{
 	8,  // 0: sisyphus.v1.LocateBlobResponse.holders:type_name -> sisyphus.v1.BlobHolder
 	15, // 1: sisyphus.v1.ListPinsResponse.pins:type_name -> sisyphus.v1.Pin
-	25, // 2: sisyphus.v1.Pin.expires_at:type_name -> google.protobuf.Timestamp
+	27, // 2: sisyphus.v1.Pin.expires_at:type_name -> google.protobuf.Timestamp
-	22, // 3: sisyphus.v1.ReplicasResponse.blobs:type_name -> sisyphus.v1.ReplicatedBlob
+	20, // 3: sisyphus.v1.ReplicateRequest.list:type_name -> sisyphus.v1.KeepList
-	0,  // 4: sisyphus.v1.BlobService.Put:input_type -> sisyphus.v1.PutBlobRequest
+	20, // 4: sisyphus.v1.ReplicateResponse.list:type_name -> sisyphus.v1.KeepList
-	2,  // 5: sisyphus.v1.BlobService.Get:input_type -> sisyphus.v1.GetBlobRequest
+	21, // 5: sisyphus.v1.KeepList.blobs:type_name -> sisyphus.v1.KeptBlob
-	4,  // 6: sisyphus.v1.BlobService.Stat:input_type -> sisyphus.v1.StatBlobRequest
+	27, // 6: sisyphus.v1.KeptBlob.keep_until:type_name -> google.protobuf.Timestamp
-	6,  // 7: sisyphus.v1.BlobService.Locate:input_type -> sisyphus.v1.LocateBlobRequest
+	24, // 7: sisyphus.v1.ReplicasResponse.blobs:type_name -> sisyphus.v1.ReplicatedBlob
-	9,  // 8: sisyphus.v1.BlobService.Pin:input_type -> sisyphus.v1.PinBlobRequest
+	0,  // 8: sisyphus.v1.BlobService.Put:input_type -> sisyphus.v1.PutBlobRequest
-	11, // 9: sisyphus.v1.BlobService.Unpin:input_type -> sisyphus.v1.UnpinBlobRequest
+	2,  // 9: sisyphus.v1.BlobService.Get:input_type -> sisyphus.v1.GetBlobRequest
-	13, // 10: sisyphus.v1.BlobService.ListPins:input_type -> sisyphus.v1.ListPinsRequest
+	4,  // 10: sisyphus.v1.BlobService.Stat:input_type -> sisyphus.v1.StatBlobRequest
-	16, // 11: sisyphus.v1.BlobService.CollectGarbage:input_type -> sisyphus.v1.CollectGarbageRequest
+	6,  // 11: sisyphus.v1.BlobService.Locate:input_type -> sisyphus.v1.LocateBlobRequest
-	18, // 12: sisyphus.v1.BlobService.Replicate:input_type -> sisyphus.v1.ReplicateRequest
+	9,  // 12: sisyphus.v1.BlobService.Pin:input_type -> sisyphus.v1.PinBlobRequest
-	20, // 13: sisyphus.v1.BlobService.Replicas:input_type -> sisyphus.v1.ReplicasRequest
+	11, // 13: sisyphus.v1.BlobService.Unpin:input_type -> sisyphus.v1.UnpinBlobRequest
-	23, // 14: sisyphus.v1.BlobService.Restore:input_type -> sisyphus.v1.RestoreRequest
+	13, // 14: sisyphus.v1.BlobService.ListPins:input_type -> sisyphus.v1.ListPinsRequest
-	1,  // 15: sisyphus.v1.BlobService.Put:output_type -> sisyphus.v1.PutBlobResponse
+	16, // 15: sisyphus.v1.BlobService.CollectGarbage:input_type -> sisyphus.v1.CollectGarbageRequest
-	3,  // 16: sisyphus.v1.BlobService.Get:output_type -> sisyphus.v1.GetBlobResponse
+	18, // 16: sisyphus.v1.BlobService.Replicate:input_type -> sisyphus.v1.ReplicateRequest
-	5,  // 17: sisyphus.v1.BlobService.Stat:output_type -> sisyphus.v1.StatBlobResponse
+	22, // 17: sisyphus.v1.BlobService.Replicas:input_type -> sisyphus.v1.ReplicasRequest
-	7,  // 18: sisyphus.v1.BlobService.Locate:output_type -> sisyphus.v1.LocateBlobResponse
+	25, // 18: sisyphus.v1.BlobService.Restore:input_type -> sisyphus.v1.RestoreRequest
-	10, // 19: sisyphus.v1.BlobService.Pin:output_type -> sisyphus.v1.PinBlobResponse
+	1,  // 19: sisyphus.v1.BlobService.Put:output_type -> sisyphus.v1.PutBlobResponse
-	12, // 20: sisyphus.v1.BlobService.Unpin:output_type -> sisyphus.v1.UnpinBlobResponse
+	3,  // 20: sisyphus.v1.BlobService.Get:output_type -> sisyphus.v1.GetBlobResponse
-	14, // 21: sisyphus.v1.BlobService.ListPins:output_type -> sisyphus.v1.ListPinsResponse
+	5,  // 21: sisyphus.v1.BlobService.Stat:output_type -> sisyphus.v1.StatBlobResponse
-	17, // 22: sisyphus.v1.BlobService.CollectGarbage:output_type -> sisyphus.v1.CollectGarbageResponse
+	7,  // 22: sisyphus.v1.BlobService.Locate:output_type -> sisyphus.v1.LocateBlobResponse
-	19, // 23: sisyphus.v1.BlobService.Replicate:output_type -> sisyphus.v1.ReplicateResponse
+	10, // 23: sisyphus.v1.BlobService.Pin:output_type -> sisyphus.v1.PinBlobResponse
-	21, // 24: sisyphus.v1.BlobService.Replicas:output_type -> sisyphus.v1.ReplicasResponse
+	12, // 24: sisyphus.v1.BlobService.Unpin:output_type -> sisyphus.v1.UnpinBlobResponse
-	24, // 25: sisyphus.v1.BlobService.Restore:output_type -> sisyphus.v1.RestoreResponse
+	14, // 25: sisyphus.v1.BlobService.ListPins:output_type -> sisyphus.v1.ListPinsResponse
-	15, // [15:26] is the sub-list for method output_type
+	17, // 26: sisyphus.v1.BlobService.CollectGarbage:output_type -> sisyphus.v1.CollectGarbageResponse
-	4,  // [4:15] is the sub-list for method input_type
+	19, // 27: sisyphus.v1.BlobService.Replicate:output_type -> sisyphus.v1.ReplicateResponse
-	4,  // [4:4] is the sub-list for extension type_name
+	23, // 28: sisyphus.v1.BlobService.Replicas:output_type -> sisyphus.v1.ReplicasResponse
-	4,  // [4:4] is the sub-list for extension extendee
+	26, // 29: sisyphus.v1.BlobService.Restore:output_type -> sisyphus.v1.RestoreResponse
-	0,  // [0:4] is the sub-list for field type_name
+	19, // [19:30] is the sub-list for method output_type
+	8,  // [8:19] is the sub-list for method input_type
+	8,  // [8:8] is the sub-list for extension type_name
+	8,  // [8:8] is the sub-list for extension extendee
+	0,  // [0:8] is the sub-list for field type_name
 }
 
 func init() { file_sisyphus_v1_blob_proto_init() }
@@ -1392,7 +1622,7 @@ func file_sisyphus_v1_blob_proto_init() {
 			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
 			RawDescriptor: unsafe.Slice(unsafe.StringData(file_sisyphus_v1_blob_proto_rawDesc), len(file_sisyphus_v1_blob_proto_rawDesc)),
 			NumEnums:      0,
-			NumMessages:   25,
+			NumMessages:   27,
 			NumExtensions: 0,
 			NumServices:   1,
 		},

packages/protocol/sisyphus/v1/blob_grpc.pb.go

@@ -63,13 +63,20 @@ type BlobServiceClient interface {
 	// and is told what it should: it fetches what it lacks with Get and drops
 	// the rest. Asking is also how the node knows the follower is there, so a
 	// follower asks again every so often.
+	//
+	// What the node tells a follower to hold it signs, and the follower keeps
+	// the signed list. A node that has lost its store asks to be shown the
+	// list, checks that the signature is its own, and takes back what the
+	// list names. A follower's word alone brings nothing back.
 	Replicate(ctx context.Context, in *ReplicateRequest, opts ...grpc.CallOption) (*ReplicateResponse, error)
 	// Replicas reports which followers hold each blob the node has pinned,
 	// against how many should.
 	Replicas(ctx context.Context, in *ReplicasRequest, opts ...grpc.CallOption) (*ReplicasResponse, error)
-	// Restore is for a node whose store was lost and replaced. It fetches
+	// Restore is for a node whose store was lost and replaced, and which
-	// back what its followers still hold from the store it had before and
+	// cannot take back by itself what its followers hold from the store it
-	// pins it on behalf of the node's user.
+	// had before: because it has lost its key too, or because a follower has
+	// no list it signed. It fetches those blobs back on the followers' word
+	// and pins them on behalf of the node's user.
 	Restore(ctx context.Context, in *RestoreRequest, opts ...grpc.CallOption) (*RestoreResponse, error)
 }
 
@@ -234,13 +241,20 @@ type BlobServiceServer interface {
 	// and is told what it should: it fetches what it lacks with Get and drops
 	// the rest. Asking is also how the node knows the follower is there, so a
 	// follower asks again every so often.
+	//
+	// What the node tells a follower to hold it signs, and the follower keeps
+	// the signed list. A node that has lost its store asks to be shown the
+	// list, checks that the signature is its own, and takes back what the
+	// list names. A follower's word alone brings nothing back.
 	Replicate(context.Context, *ReplicateRequest) (*ReplicateResponse, error)
 	// Replicas reports which followers hold each blob the node has pinned,
 	// against how many should.
 	Replicas(context.Context, *ReplicasRequest) (*ReplicasResponse, error)
-	// Restore is for a node whose store was lost and replaced. It fetches
+	// Restore is for a node whose store was lost and replaced, and which
-	// back what its followers still hold from the store it had before and
+	// cannot take back by itself what its followers hold from the store it
-	// pins it on behalf of the node's user.
+	// had before: because it has lost its key too, or because a follower has
+	// no list it signed. It fetches those blobs back on the followers' word
+	// and pins them on behalf of the node's user.
 	Restore(context.Context, *RestoreRequest) (*RestoreResponse, error)
 	mustEmbedUnimplementedBlobServiceServer()
 }

proto/sisyphus/v1/blob.proto

@@ -38,13 +38,20 @@ service BlobService {
   // and is told what it should: it fetches what it lacks with Get and drops
   // the rest. Asking is also how the node knows the follower is there, so a
   // follower asks again every so often.
+  //
+  // What the node tells a follower to hold it signs, and the follower keeps
+  // the signed list. A node that has lost its store asks to be shown the
+  // list, checks that the signature is its own, and takes back what the
+  // list names. A follower's word alone brings nothing back.
   rpc Replicate(ReplicateRequest) returns (ReplicateResponse);
   // Replicas reports which followers hold each blob the node has pinned,
   // against how many should.
   rpc Replicas(ReplicasRequest) returns (ReplicasResponse);
-  // Restore is for a node whose store was lost and replaced. It fetches
+  // Restore is for a node whose store was lost and replaced, and which
-  // back what its followers still hold from the store it had before and
+  // cannot take back by itself what its followers hold from the store it
-  // pins it on behalf of the node's user.
+  // had before: because it has lost its key too, or because a follower has
+  // no list it signed. It fetches those blobs back on the followers' word
+  // and pins them on behalf of the node's user.
   rpc Restore(RestoreRequest) returns (RestoreResponse);
 }
 
@@ -111,7 +118,8 @@ message Pin {
   string cid = 1;
   // Who holds the pin: "user", "job:<id>", or "recent" for the short pin
   // every newly stored blob gets. In a follower's store of copies it is
-  // "pool:" and the ID of the store they are copies of.
+  // "pool:" and the ID of the store they are copies of. "restored" is on
+  // what a node that lost its store took back from its followers.
   string owner = 2;
   // Unset for a pin that holds until released.
   google.protobuf.Timestamp expires_at = 3;
@@ -131,13 +139,54 @@ message ReplicateRequest {
   // The store those are copies of, as the node named it in its last answer.
   // Empty for a follower that holds nothing.
   string store = 2;
+  // The last list the node signed for this follower, handed back when the
+  // node asked to be shown it.
+  KeepList list = 3;
+  // Set by a follower that keeps the node's signed lists. It is then told
+  // what to hold in a list alone, and hold is left empty.
+  bool keeps_lists = 4;
 }
 
 message ReplicateResponse {
   // Everything the follower should hold from now on, and nothing else.
+  // Empty if list is set: the list then says the same.
   repeated string hold = 1;
   // What to give as the store next time.
   string store = 2;
+  // What the follower should hold, signed, for it to keep in place of the
+  // list it had. Unset while the follower holds blobs from an earlier store
+  // that the node has not taken back: the list it has is then the only
+  // proof of what that store kept.
+  KeepList list = 3;
+  // Set when the follower holds blobs from a store other than the node's
+  // present one and gave no list. It should ask again at once, with the
+  // list it has.
+  bool show_list = 4;
+}
+
+// KeepList is a node's statement of what one follower is to hold, signed so
+// that the node can later tell it for its own.
+message KeepList {
+  // The store the list is of: the node's ID, "/", and the store's name.
+  string store = 1;
+  // When the node signed the list, in nanoseconds since 1970, or one more
+  // than its last if its clock had not moved on. It only goes up.
+  uint64 sequence = 2;
+  repeated KeptBlob blobs = 3;
+  // The node's signature, with its node key, of this encoding of the fields
+  // above, in which every number is eight bytes, most significant first:
+  // the text "sisyphus keep list v1" and a newline; the length of store and
+  // store; the sequence; the number of blobs; and for each blob the length
+  // of its CID, the CID, and either a zero byte, if it has no keep_until,
+  // or a one byte and keep_until in nanoseconds since 1970.
+  bytes signature = 4;
+}
+
+message KeptBlob {
+  string cid = 1;
+  // The latest expiry of the node's pins on the blob. Unset if one of them
+  // holds until released.
+  google.protobuf.Timestamp keep_until = 2;
 }
 
 message ReplicasRequest {
@@ -156,8 +205,16 @@ message ReplicasResponse {
   // before are given time to say so first.
   bool settling = 4;
   // How many blobs followers hold from a store the node had before and
-  // has not pinned in the one it has now. Restore takes them back.
+  // has not pinned in the one it has now.
   uint32 from_earlier_store = 5;
+  // Set while the node is taking back what a list it signed names.
+  bool restoring = 6;
+  // How many blobs the node has pinned as "restored": those it took back
+  // by itself and is still keeping.
+  uint32 restored = 7;
+  // How many of from_earlier_store no list signed by the node covers. The
+  // node does not take these back by itself; Restore does.
+  uint32 unsigned_from_earlier_store = 8;
 }
 
 message ReplicatedBlob {