Distributed Coordination

Using Forum across peers to coordinate work between agents on different machines.

What this page covers

Forum is local-first. Cross-process coordination works through Strand replication. This page explains the model, the convergence behavior, and the pitfalls.

Each Forum wraps a Strand. Two peers that share the same Strand (via replication) see the same TupleEntry::Write and TupleEntry::Take blocks eventually. Each peer maintains its own live_tuples in-memory index.

   Peer A                                  Peer B
   ──────                                  ──────
   Forum::write(b"job-1")                  
        │                                  
        ▼                                  
   strand.append(Write{...})               
        │                                  
        │  Strand replication              
        │  (weave-dht / weave-swarm)       
        └────────────────────────────────► strand.append(Write{...})
                                                   │
                                                   ▼
                                           live_tuples on B sees
                                           the new tuple after
                                           the next rebuild pass

The race that matters

Warning

Two peers can both call take for the same pattern before each peer has replicated the other's Take tombstone. Both takes return successfully on their local indices. After replication, both peers will see two Take entries against the same Write — the second one is a no-op in terms of removed tuples, but the consumer on the slower peer has already started work.

If your workflow cannot tolerate double-consumption, use one of these patterns:

Pattern 1: Coordinator-claims with a Lens

Add a claim step that writes the claim into a Lens database keyed by tuple id. The Lens conflict resolution makes the claim decidable: only the first claim wins after convergence.

use weave_sdk::prelude::*;

async fn claim_job(node: &WeaveNode, tuple_id: u64, worker: &str) -> WeaveResult<bool> {
    let key = tuple_id.to_be_bytes();
    if node.lens_get("claims", &key).await?.is_some() {
        return Ok(false);
    }
    node.lens_put("claims", &key, worker.as_bytes()).await?;
    Ok(true)
}

Pattern 2: Partition the tuple space

Encode the worker id (or a hash of it) into the tuple kind. Each worker only takes tuples addressed to it. No two workers ever see the same tuple.

use forum::Pattern;

fn pattern_for_worker(worker_id: &str, kind: &str) -> Pattern {
    Pattern(format!("{worker_id}/{kind}").into_bytes())
}

Pattern 3: Leader-only writes

Have one peer be the canonical writer; everyone else is read-only. This is the right shape for fanout queues where many readers consume from one producer.

Replication latency expectations

Strand replication runs on the weave-dht/weave-swarm paths. Round-trip latency between two peers is bounded by:

T_replication ≈ RTT_network + T_disk_flush + T_index_rebuild

For LAN peers with SSD storage, this is typically tens of milliseconds. For WAN peers, plan on hundreds of milliseconds.

Convergence guarantees

Property	Guarantee
Eventual delivery of `Write` blocks	Yes, under the standard Strand replication assumptions
Eventual delivery of `Take` blocks	Yes
Same final tuple set on all peers	Yes
Same intermediate `live_tuples` map across peers	No
Same order of `EntryId` assignments	Yes (Strand is per-writer total-ordered)

A practical multi-peer example

use weave_sdk::prelude::*;

#[tokio::main]
async fn main() -> WeaveResult<()> {
    let node = WeaveNode::builder()
        .namespace("l1fe")
        .identifier("worker-a")
        .storage_dir("/tmp/weave-worker-a")
        .build()
        .await?;

    node.start_network().await?;
    node.start_auto_replication().await?;

    node.open_forum("jobs").await?;

    // Producer side.
    node.forum_write("jobs", b"render frame 42".to_vec()).await?;

    // Consumer side (different process, same strand via replication).
    if let Some(payload) = node.forum_take("jobs", b"render frame 42").await? {
        println!("worker-a got {}", String::from_utf8_lossy(&payload));
    }

    Ok(())
}

Run two copies of this binary with different identifiers and shared connectivity. Only one of them will receive the Some branch for any given Write — modulo the race described above.