Fetch

Reconstruct a Weft payload from chunks served by one or more peers, verifying against the manifest root.

What this page covers

The consumer-side flow: given a root (and access to a manifest), fetch the chunks, verify them against the manifest, and assemble the original bytes.

The fetch model

A fetch needs three things:

A WeftManifest — proves the structure and chunk ids of the payload.
A ChunkSource — knows how to retrieve a chunk by id from one or more peers.
Availability information — a map from chunk index to the peers that have it.

The fetch_manifest function performs the work, verifying each chunk's BLAKE3 id before accepting it.

Note

Every chunk is verified against its BLAKE3 id before delivery. If a peer serves bytes that don't match the id, the response is discarded and the fetcher tries another peer from the availability list — corrupted or adversarial peers cannot poison a fetch.

Local fetch (one peer, in-process)

The simplest case: a consumer in the same process as the producer.

use weft::{
    chunk::{ChunkConfig, DEFAULT_CHUNK_SIZE},
    discovery::PeerId,
    local::LocalWeftStore,
    manifest::{ArtifactMetadata, ManifestProducer},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = LocalWeftStore::in_memory();
    let payload = b"the quick brown fox jumps over the lazy dog".to_vec();

    let producer = ManifestProducer {
        did: "did:oas:agent:demo".into(),
        signing_key_id: None,
        signature: vec![0u8; 64],
    };
    let artifact = ArtifactMetadata {
        artifact_id:          None,
        media_type:           "text/plain".into(),
        schema:               "weave.text.v1".into(),
        mars_asset_id:        None,
        sigil_commitment_cid: None,
    };

    let published = store.publish_bytes(
        &payload,
        ChunkConfig::new(DEFAULT_CHUNK_SIZE)?,
        producer,
        artifact,
        PeerId::new("did:peer:demo")?,
    )?;

    // Consumer side: given just the root, fetch the bytes back.
    let max_parallelism = 4;
    let bytes = store.fetch_payload(published.manifest.root, max_parallelism).await?;
    assert_eq!(bytes, payload);

    Ok(())
}

LocalWeftStore::fetch_payload builds a FetchRequest internally, calls fetch_manifest, and returns the assembled bytes.

Fetch through a custom ChunkSource

Real network fetches plug in a ChunkSource implementation:

use async_trait::async_trait;
use std::sync::Arc;
use weft::discovery::PeerId;
use weft::fetch::{fetch_manifest, ChunkSource, FetchRequest, InMemoryChunkSource};
use weft::{WeftHash, Result};

#[async_trait]
trait MyChunkAdapter: Send + Sync {
    async fn fetch(&self, peer: &PeerId, id: WeftHash) -> std::result::Result<Vec<u8>, String>;
}

struct AdapterSource<A: MyChunkAdapter> {
    adapter: A,
}

#[async_trait]
impl<A: MyChunkAdapter + 'static> ChunkSource for AdapterSource<A> {
    async fn fetch(&self, peer: &PeerId, id: WeftHash) -> Result<Vec<u8>> {
        self.adapter
            .fetch(peer, id)
            .await
            .map_err(|e| weft::WeftError::Fetch(e))
    }
}

The ChunkSource trait is the entire surface a network adapter must implement. The fetch engine in fetch_manifest does the rest: scheduling, verification, retry on WeftError::Hash mismatches.

What `fetch_manifest` does

fetch_manifest(request, source)
   │
   ├─► For each chunk index 0..chunk_count:
   │     candidates = request.availability.get(chunk_index)
   │     for each candidate peer in candidates:
   │       try:
   │         bytes = source.fetch(peer, chunk_id).await?
   │         verified_id = blake3(domain || bytes)
   │         if verified_id != chunk_id: continue   // try next peer
   │         buffer.extend(bytes)
   │         break
   │     else:
   │       return Err(no peer served chunk index)
   │
   └─► Ok(buffer)

The max_parallelism field in FetchRequest bounds in-flight chunk fetches. Verification happens after each chunk lands, so a malicious peer cannot inject content under a wrong id.

Errors

Error	Cause	Recovery
`WeftError::InvalidHash`	Returned bytes did not hash to the expected chunk id	Try the next advertised peer
`WeftError::Fetch(String)`	Network adapter failed	Retry with backoff
`WeftError::Manifest("...")`	Manifest itself failed validation	The manifest is the wrong file; re-fetch from the producer
`WeftError::Unsupported(...)`	`ParitySpec` set, or DHT adapter not linked	Disable parity in publishing; link a DHT adapter

Tuning fetches

Variable	What it controls
`FetchRequest::max_parallelism`	Concurrent chunk fetches
Availability map size	Peer choice; more peers means more redundancy
Chunk size on the producer side	Bigger chunks → fewer round trips, more bytes per retry

Practical patterns

Resumable fetch

The LocalWeftStore and WeftDhtStore both keep chunks once stored. A second fetch_payload call uses the cached chunks first. To resume an interrupted fetch, point the same store at the same root and call fetch_payload again — already-fetched chunks are skipped.

Verifying before serving

If you are a downstream peer mirroring an artifact, you should fetch and verify before serving. Both stores do this automatically: stored chunks carry their canonical id as the key, and the chunk id is recomputed from the bytes on insert.

On this page