Semantic Search

What you'll build

A small ingestion pipeline that takes a folder of text documents, embeds them, stores embeddings in Basis, stores the source text in a Strand, and records authorship metadata in Gnosis. A second command runs a semantic search and resolves the hits back to text + metadata.

Time: 25 minutes.

Primitives used: Basis (HNSW vector index), Strand (signed document store), Gnosis (metadata triples).

Prerequisites

• Rust 1.78+.
• A text-embedding model. The example uses a deterministic stub so the tutorial runs without network access; swap in a real embedder before production use.

Build the pipeline

1. 1
   Scaffold the project

   cargo new --bin semantic-search
   cd semantic-search

2. 2
   Declare dependencies

   [package]
   name = "semantic-search"
   version = "0.1.0"
   edition = "2021"
   
   [dependencies]
   weave-sdk = "1.1.0"
   tokio     = { version = "1", features = ["rt-multi-thread", "macros"] }
   anyhow    = "1"
   serde     = { version = "1", features = ["derive"] }
   serde_json = "1"
   uuid      = { version = "1", features = ["v4"] }
   blake3    = "1"

3. 3
   Write the ingestion + search node

   use anyhow::Result;
   use serde::{Deserialize, Serialize};
   use std::env;
   use uuid::Uuid;
   use weave_sdk::prelude::*;
   
   const EMBED_DIM: usize = 64; // small for the deterministic stub
   
   #[derive(Serialize, Deserialize, Clone)]
   struct Doc {
       id:     Uuid,
       title:  String,
       body:   String,
       author: String,
   }
   
   #[tokio::main]
   async fn main() -> Result<()> {
       let cmd = env::args().nth(1).unwrap_or_else(|| "help".into());
       let node = WeaveNode::builder()
           .namespace("rag-demo")
           .identifier("rag-node")
           .storage_dir("/tmp/rag-demo")
           .build()
           .await?;
   
       node.open_basis("docs-vec").await?;
       node.create_strand("docs-text").await.ok(); // tolerate exists
       node.open_gnosis("docs-meta").await?;
   
       match cmd.as_str() {
           "ingest" => {
               let title  = env::args().nth(2).unwrap_or_else(|| "untitled".into());
               let author = env::args().nth(3).unwrap_or_else(|| "unknown".into());
               let body   = read_stdin().await?;
               ingest(&node, &title, &author, &body).await?;
           }
           "search" => {
               let query = env::args().nth(2).unwrap_or_else(|| "weave".into());
               let k     = env::args().nth(3).and_then(|s| s.parse().ok()).unwrap_or(5);
               search(&node, &query, k).await?;
           }
           _ => {
               eprintln!("usage:");
               eprintln!("  ingest <title> <author>   # body on stdin");
               eprintln!("  search <query> [k]");
           }
       }
   
       Ok(())
   }
   
   async fn ingest(node: &WeaveNode, title: &str, author: &str, body: &str) -> Result<()> {
       let id = Uuid::new_v4();
       let doc = Doc {
           id,
           title:  title.to_string(),
           body:   body.to_string(),
           author: author.to_string(),
       };
   
       // 1) Append the canonical document to the Strand.
       let bytes = serde_json::to_vec(&doc)?;
       let seq = node.append("docs-text", &bytes).await?;
   
       // 2) Embed and index.
       let emb = embed(body);
       node.basis_add("docs-vec", id, &emb).await?;
   
       // 3) Metadata triples.
       let id_str = id.to_string();
       node.gnosis_add("docs-meta", &id_str, "has_title",   title).await?;
       node.gnosis_add("docs-meta", &id_str, "authored_by", author).await?;
       node.gnosis_add("docs-meta", &id_str, "strand_seq",  &seq.to_string()).await?;
   
       println!("ingested {id} (strand seq {seq}, embedding dim {})", emb.len());
       Ok(())
   }
   
   async fn search(node: &WeaveNode, query: &str, k: usize) -> Result<()> {
       let q_emb = embed(query);
   
       let hits = node.basis_search("docs-vec", &q_emb, k).await?;
       if hits.is_empty() {
           println!("no matches");
           return Ok(());
       }
   
       println!("top {k} matches:");
       for (id, distance) in hits {
           // Resolve author and title via Gnosis.
           let id_str = id.to_string();
           let title = node
               .gnosis_query("docs-meta", Some(&id_str), Some("has_title"), None)
               .await?
               .into_iter()
               .next()
               .map(|t| t.object)
               .unwrap_or_else(|| "(unknown title)".into());
           let author = node
               .gnosis_query("docs-meta", Some(&id_str), Some("authored_by"), None)
               .await?
               .into_iter()
               .next()
               .map(|t| t.object)
               .unwrap_or_else(|| "(unknown)".into());
   
           println!("  {distance:.4}  {title}  ({author})  [{id}]");
       }
   
       Ok(())
   }
   
   /// Toy deterministic embedding for the tutorial. Replace with a real model
   /// (OpenAI embeddings, sentence-transformers, etc.) for any non-demo use.
   fn embed(text: &str) -> Vec<f32> {
       let mut hasher = blake3::Hasher::new();
       hasher.update(text.as_bytes());
       let hash = hasher.finalize();
       let bytes = hash.as_bytes();
   
       let mut out = vec![0.0f32; EMBED_DIM];
       for (i, slot) in out.iter_mut().enumerate() {
           let b = bytes[i % bytes.len()];
           *slot = (b as f32 / 255.0) * 2.0 - 1.0; // [-1, 1]
       }
       // Normalize so L2 nearest matches cosine nearest.
       let norm: f32 = out.iter().map(|x| x * x).sum::<f32>().sqrt();
       if norm > 0.0 {
           for x in &mut out { *x /= norm; }
       }
       out
   }
   
   async fn read_stdin() -> Result<String> {
       use tokio::io::AsyncReadExt;
       let mut buf = String::new();
       tokio::io::stdin().read_to_string(&mut buf).await?;
       Ok(buf)
   }

4. 4
   Ingest documents

   echo "Weave is a P2P stack for agent-native apps." | cargo run -- ingest weave-intro alice
   echo "The Sigil chain provides anchored commitments." | cargo run -- ingest sigil-intro bob
   echo "Weft distributes large blobs over the DHT."     | cargo run -- ingest weft-intro alice

5. 5
   Run a query

   cargo run -- search "blob distribution" 3

   top 3 matches:
     0.0142  weft-intro  (alice)  [...]
     0.0871  weave-intro (alice)  [...]
     0.1102  sigil-intro (bob)    [...]

How the primitives compose

• Strand is the canonical document store. Each document is a signed block; the strand sequence number is the durable id.
• Basis is the embedding index. Adding to Basis writes a VectorEntry::Add block to Basis's internal strand and inserts the vector into HNSW.
• Gnosis is the metadata layer. Triples like (uuid, authored_by, alice) give us fast lookups by author without scanning the document Strand.

Note the shared id space: the same Uuid keys the Basis entry, names the Gnosis subject, and is recoverable from the Strand block. No primitive owns the id; it threads through all three.

Production-readiness

This tutorial uses a deterministic toy embedder. For real applications:

What to do next

• Add chunk-level retrieval by splitting documents and indexing each chunk separately.
• Use Weft to distribute large reference corpora to other peers.
• Combine with Social Feed to index a real-time feed and search it.
• See Basis: insert and search for the underlying parameters.