Fetch and Parse

Purpose

Agent browser engine for fetch, parse, search, perceive, research, protocol routing, and HTTP serving.

This page follows the real source shape for Weave Browser Engine and explains the workflow a developer is likely to use first.

Developer workflow

Start from the smallest constructor or builder, perform one meaningful operation, inspect the returned state, then add the relevant policy, storage, or network integration. The examples below should be expanded whenever the crate API changes.

Primary types to know

• DuckDuckGoEngine — libs/weave-browser-engine/src/search.rs
• FetchOptions — libs/weave-browser-engine/src/fetch.rs
• FetchRequest — libs/weave-browser-engine/src/serve.rs
• HnsResolver — libs/weave-browser-engine/src/protocol.rs
• HttpResolver — libs/weave-browser-engine/src/protocol.rs
• HttpsResolver — libs/weave-browser-engine/src/protocol.rs
• LocalPerceptionBackend — libs/weave-browser-engine/src/perceive.rs
• PerceiveRequest — libs/weave-browser-engine/src/serve.rs
• ProtocolRouter — libs/weave-browser-engine/src/protocol.rs
• ResearchConfig — libs/weave-browser-engine/src/research.rs
• ResearchRequest — libs/weave-browser-engine/src/serve.rs
• SearchRequest — libs/weave-browser-engine/src/serve.rs

Example shape

use weave_browser_engine::fetch::{fetch_url, FetchOptions};
use weave_browser_engine::parse::{extract_links, extract_text};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // fetch_url returns a JSON envelope (`content_type`, `body`, `text`, etc.)
    // so callers can branch on type without parsing again.
    let options = FetchOptions {
        timeout_ms: 5_000,
        user_agent: Some("weave-doc/0.1".into()),
        ..FetchOptions::default()
    };
    let response = fetch_url("https://example.com", &options).await?;

    // The parse module exposes ad-hoc extraction primitives that operate on
    // raw HTML — useful when callers want to skip the fetch JSON envelope.
    let html = response["text"].as_str().unwrap_or("");
    let links = extract_links(html);
    let preview = extract_text(html).chars().take(120).collect::<String>();

    println!(
        "content_type={} link_count={} preview={}",
        response["content_type"].as_str().unwrap_or(""),
        links.len(),
        preview,
    );
    Ok(())
}