Failure Modes What happens to each Weave primitive under node crashes, peer disconnects, storage exhaustion, replication conflicts, and clock skew — and how to recover.
A per-primitive crash-and-recovery table. For each failure scenario, what state survives, what is lost, and what the operator (or the application) needs to do.
Every primitive in Weave follows the same general recovery shape: open performs replay or rebuild from a verifiable source-of-truth (Strand), and on-read hash checks catch corruption that survived replay.
Scenario Definition Process crash The Weave node is terminated without graceful shutdown Disk full Local storage runs out of space mid-write Peer disconnect A remote peer becomes unreachable mid-replication Replication conflict Two peers attempt incompatible writes that race Clock skew Local wall-clock disagrees with peer wall-clock Data corruption A storage block is unreadable or fails its hash check
Scenario What survives What is lost Recovery Process crash mid-append Blocks that returned Ok from append The in-flight block On open, the WAL is replayed; partial blocks are discarded Disk full Existing blocks remain readable The block that triggered the error Free disk, retry; no rollback needed Peer disconnect Local replica is intact The blocks that had not yet replicated Reconnect; replication catches up using the Merkle bitfield Replication conflict Each writer's strand is internally consistent Nothing — writers do not share strands by design Use Nexus to combine multiple writers' strands Clock skew Strand order is sequence-number based, not time Nothing None required Data corruption All blocks except the corrupted one The corrupted block Strand provides per-block hashes; corrupted blocks fail their check on read and the caller can request re-replication from a peer
Open the strand
Strand::new(config) opens the storage and runs WAL replay.
Truncate inconsistent tail
Drop any incomplete sequence at the tail; the WAL guarantees the prefix is consistent.
Mark missing blocks on read
If a block fails its hash check, mark it as missing in the bitfield and request re-replication.
Catch up via replication
Replication fills gaps from any peer holding a valid copy. Once the bitfield is full, the strand is healthy.
Scenario What survives What is lost Recovery Process crash mid-put Snapshots up to the last commit The uncommitted batch Reopen; in-flight batch is discarded Disk full Last committed snapshot The pending write Free disk, retry put Peer disconnect Local Lens is intact Pending replication progress Reconnect; Strand backing the Lens catches up Replication conflict Each writer's Lens snapshots are independent n/a (single writer per Lens) Use Nexus for multi-writer KV Data corruption Snapshots not touching the corrupted block Reads through the affected key range Run the Lens' built-in repair (see Lens internals)
Scenario What survives What is lost Recovery Process crash mid-locus_write_file Files committed before the crash The in-flight file (its metadata block may or may not have committed) On open, journal replay completes or rolls back the file write Disk full Already-written files The file that triggered the error; the metadata block may indicate it exists with zero bytes Re-write the file with the same path Peer disconnect Local mount intact Pending sync to peers Reconnect; Locus' sync engine resumes Replication conflict (two writers touching the same path) Both writers' Strands persist their intent The conflicting state is observable on both peers Locus exposes conflict markers; resolve at the application layer Data corruption Files outside the affected block range Affected files Re-fetch from a peer with a healthy replica
Scenario What survives What is lost Recovery Crash mid-materialization Upstream strands intact The materialized view's tip Re-derive the view from the upstream strands Upstream strand unavailable The view continues to serve up to the unavailable point New data from that source Reconnect to the upstream; resume from the saved offset Conflicting upstream writes All upstreams retained n/a (Nexus is deterministic given its inputs) None required; the view re-derives consistently
Scenario What survives What is lost Recovery Crash mid-take The Strand may or may not have the Take tombstone If the tombstone landed but live_tuples did not update, the next process restart restores it; if the tombstone did NOT land, the tuple appears live to other peers Worker handlers must be idempotent (see Take Tuple ) Disk full Tuples already written The tuple being written Retry the write Peer disconnect Local Forum state intact Replication of Write / Take blocks Reconnect; Strand catches up Two peers both take a tuple before the tombstone replicates Both peers' Strands persist both Take records Useful semantics — the workflow must handle double-consumption Use coordinator-claim pattern (see Distributed Coordination )
Warning
Forum take is not a distributed mutex. Two peers can both take a tuple before the tombstone replicates, and both take calls will return Some. Worker handlers must be idempotent or use the coordinator-claim pattern.
Scenario What survives What is lost Recovery Crash before index update Triple is in the Strand The in-memory index entry On restart, rebuild indices from the Strand (the SDK does this on open_gnosis) Disk full Triples already committed The triple being added Retry Index corruption The Strand The three DashMaps Rebuild from the Strand
Scenario What survives What is lost Recovery Crash before HNSW update Vector is in the Strand The HNSW node On restart, rebuild_index is needed (today this requires a remove of a present id; future: explicit compact()) Disk full Existing vectors The pending insert Retry Memory exhaustion The vector Strand Search ability Shard the corpus across multiple Basis instances on different processes Index strand corruption No real impact today Nothing (the index strand is a stub) Delete and recreate
Scenario What survives What is lost Recovery Crash mid-publish Chunks that hit disk The publish is incomplete; the manifest may not exist yet Re-publish; chunk ids are deterministic so duplicates are silently merged Disk full during chunk store Already-stored chunks The chunk that triggered the error Free disk, re-publish All peers serving a chunk disappear Manifest still valid (proves the root) The chunk's bytes are unrecoverable from the network Re-publish from a producer who has the source bytes Chunk fails hash verification on fetch Other chunks intact The malicious peer's response is discarded The fetcher tries another peer from the availability list Manifest signature invalid n/a The manifest is rejected Re-fetch from the producer or a trusted mirror
Scenario What survives What is lost Recovery Node crash Peers' replicas of records the node hosted This node's view of the DHT Re-join; bootstrap from peers; refetch needed records Network partition Each partition's records Cross-partition reachability Heals automatically on reconnection; queries during the partition may fail Eclipse (adversary controls all neighbors) Local data Useful DHT lookups Mitigation: use α = 3 parallel queries (default), monitor for unusual neighbor turnover
Scenario What survives What is lost Recovery Connection death Other connections The disconnected peer The swarm reattempts; pending events are queued up to a bound Backpressure exhaustion Recent messages Older messages (dropped) Tune the priority queue limit
Primitive Cold start After data corruption After partition heal Strand WAL replay (automatic) Mark missing block, request re-replication Replication catches up via bitfield Lens Open snapshot (automatic) Repair via Lens internals Underlying Strand replication Locus Journal replay (automatic) Re-fetch affected files Sync engine resumes Nexus Re-derive view (automatic) Re-derive from upstreams Catches up with upstreams Forum Rebuild live_tuples from Strand Worker idempotency handles it Strand replication Gnosis Rebuild indices from Strand Discard bad blocks, continue Strand replication Basis Empty HNSW (rebuild on first remove) Use vector_strand as source of truth Strand replication Weft Reopen store; chunks are content-addressed Discard corrupted chunks; re-fetch Re-advertise availability
Signal Meaning WeaveNode::peer_count is zeroNo replication will progress Strand len() diverges between peers Replication backlog Repeated WeftError::InvalidHash from a peer Possible adversary or corrupted mirror; consider scoring Process memory growing unbounded with Basis active Vector corpus exceeded sizing; shard Lens snapshot count growing unbounded Compaction backlog; trigger manual compact Locus journal growing unbounded Journal trim is not running
Threat Model — adversarial scenarios in addition to these crash scenariosArchitecture — where each failure is contained in the stack