Internals

What this page covers

The exact state held by a Gnosis instance, the locking model, the query algorithm, and the cold-start rebuild.

State

pub struct Gnosis {
    strand: Arc<RwLock<Strand>>,
    subject_index:   Arc<DashMap<String, Vec<u64>>>,
    predicate_index: Arc<DashMap<String, Vec<u64>>>,
    object_index:    Arc<DashMap<String, Vec<u64>>>,
}

Field	Role
`strand`	Canonical append-only log of serde_json-encoded `Triple` blocks
`subject_index`	Maps `subject → [sequence_number, ...]`
`predicate_index`	Maps `predicate → [sequence_number, ...]`
`object_index`	Maps `object → [sequence_number, ...]`

DashMap is internally sharded, so concurrent inserts on different keys do not contend. Reads are lock-free per shard.

Insert path

add_triple(s, p, o)
   │
   ├─► Build Triple { subject, predicate, object }
   │   serde_json::to_vec(&triple)
   │
   ├─► strand.write().await
   │   seq = strand.append(bytes).await?
   │   release lock
   │
   ├─► subject_index.entry(s).or_default().push(seq)
   ├─► predicate_index.entry(p).or_default().push(seq)
   ├─► object_index.entry(o).or_default().push(seq)
   │
   └─► Ok(())

The Strand append is the linearization point. Index updates are observable to readers immediately after the per-key DashMap write completes — they are not transactional across the three indices.

There is a tiny window where a query could see the new sequence number in subject_index but not yet in object_index. For an exact (s, p, o) lookup, the result might briefly be missing. The same query a moment later succeeds.

Query algorithm

query(pattern { subject, predicate, object })
   │
   ├─► matching_indices: Option<HashSet<u64>> = None
   │
   ├─► if subject = Some(s):
   │     if let Some(idx) = subject_index.get(s):
   │       matching_indices := Some(set(idx))
   │     else:
   │       return Ok(empty)
   │
   ├─► if predicate = Some(p):
   │     if let Some(idx) = predicate_index.get(p):
   │       matching_indices := matching_indices ∩ idx  (or set(idx) if None)
   │     else:
   │       return Ok(empty)
   │
   ├─► if object = Some(o):
   │     if let Some(idx) = object_index.get(o):
   │       matching_indices := matching_indices ∩ idx  (or set(idx) if None)
   │     else:
   │       return Ok(empty)
   │
   ├─► if matching_indices = None:    // all three were None
   │     scan the entire strand and decode every block
   │
   └─► for each seq in matching_indices:
         data = strand.get(seq).await?
         if let Ok(triple) = serde_json::from_slice::<Triple>(&data):
           results.push(triple)

The intersection is implemented with HashSet::retain. The current implementation does not pick the smallest set first — this is a low-impact optimization for the common case where one position is much more selective than the others.

get_neighbors

get_neighbors(node)
   │
   ├─► if let Some(indices) = subject_index.get(node):
   │     for seq in indices:
   │       data = strand.get(seq).await?
   │       if let Ok(triple) = ...:
   │         push (triple.predicate, triple.object)
   │
   └─► return list

get_neighbors is exactly the "subject-only" query, post-processed into (predicate, object) pairs. Inverse neighbors (incoming edges) require an object_index query at the call site.

Cold-start rebuild

The constructor Gnosis::new(strand) builds an empty graph. The three indices are not persisted today. After a restart, walk the Strand once to rebuild:

use std::sync::Arc;
use tokio::sync::RwLock;
use dashmap::DashMap;
use strand::Strand;
use gnosis::{Gnosis, Triple};

async fn rebuild_indices(
    strand: Arc<RwLock<Strand>>,
) -> Result<(DashMap<String, Vec<u64>>,
            DashMap<String, Vec<u64>>,
            DashMap<String, Vec<u64>>), Box<dyn std::error::Error>>
{
    let s_index = DashMap::new();
    let p_index = DashMap::new();
    let o_index = DashMap::new();

    let s = strand.read().await;
    let len = s.len();
    for i in 0..len {
        let bytes = s.get(i).await?;
        if let Ok(triple) = serde_json::from_slice::<Triple>(&bytes) {
            s_index.entry(triple.subject).or_insert_with(Vec::new).push(i);
            p_index.entry(triple.predicate).or_insert_with(Vec::new).push(i);
            o_index.entry(triple.object).or_insert_with(Vec::new).push(i);
        }
    }

    Ok((s_index, p_index, o_index))
}

The weave-sdk does the equivalent work on open_gnosis. If you construct Gnosis directly, plan to walk the strand on startup.

Concurrency

Operation	Strand lock	Index locks
`add_triple`	Brief write lock during append	Per-shard `DashMap` writes (no global lock)
`query`	Multiple read locks during result fetch	Lock-free reads from `DashMap`
`get_neighbors`	Read lock for fetched blocks	Lock-free read

The system handles many concurrent readers and a single writer well. For many concurrent writers, the Strand append is the bottleneck; consider partitioning by subject into multiple Gnosis instances.

Cost and sizing

Footprint	Bound
Strand bytes	Sum of `serde_json(Triple)` for every fact
Index entries	`3 × N` `Vec<u64>` slots, sharing 8 B each
Total memory	~ `3 × N × 8 + sum(string keys)` bytes

For 1 M facts with average key length of 32 B: ~144 MB index + JSON-encoded Strand.

Failure injection points

Crash between Strand append and index update. On restart, the rebuild pass picks up the triple from the Strand and the indices are correct again.
Corrupt one Strand block. The rebuild path uses if let Ok(triple) = serde_json::from_slice so a bad block is silently skipped — the surrounding triples remain queryable.
Many concurrent writers. Strand serializes them; throughput is bounded by Strand append latency.

Where to extend

Pick the smallest matching set first during query intersection.
Persist index entries to a Lens to avoid the full Strand replay on startup.
Add a streaming query_stream that yields triples without buffering.
Add typed predicate constraints (e.g. integer-valued objects with range queries).

On this page