Skip to content
AlessandroS01Public

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

ย 

History

36 Commits

Folders and files

NameName
Last commit message
Last commit date
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

โšก Strata

100% Offline, Air-Gapped Personal Knowledge Engine & Local Hybrid RAG Architecture

Python 3.13+ uv SQLite Qdrant BM25 FastEmbed FlashRank Ollama Ruff Mypy Strict Pytest License: MIT


๐Ÿ“– Overview

Strata is an air-gapped, privacy-preserving personal knowledge engine and hybrid retrieval-augmented generation (RAG) system built from first principles.

Most conventional RAG prototypes treat vector databases as primary data stores. When embedding models change, vectors get corrupted, or text is updated, these systems either fail or require expensive, full-collection re-embedding.

Strata solves this by enforcing an uncompromising architectural boundary:

  • SQLite is the Canonical Source of Truth: Documents, structural Markdown sections, tokenized search chunks, and cryptographic hashes live exclusively in an ACID-compliant SQLite store.
  • Qdrant and BM25 are Ephemeral Derived Indices: Both dense vectors (embedded Qdrant) and sparse inverted indices (BM25) are strictly disposable. If either index is deleted or wiped, running strata sync deterministically reconstructs them directly from SQLite.
  • Content-Addressable Differential Invalidation: Every chunk is fingerprinted with a SHA-256 hash over its heading breadcrumb path and text body (heading_path:text). Unchanged chunks bypass embedding computation entirely ($0\text{ ms}$ overhead), while newly created chunks are incrementally embedded and orphaned chunks are purged.
  • Decoupled Sub-200ms Hybrid Search: The search engine executes parallel dense semantic search, sparse lexical BM25 matching, Reciprocal Rank Fusion (RRF), and cross-encoder re-ranking in under 200 ms without loading or invoking an LLM.
  • Hierarchical Small-to-Big Retrieval: Small child chunks (~350 tokens) are indexed for precise semantic matching, but automatically hydrate their parent section context (~900+ tokens) and adjacent sibling windows from SQLite prior to presentation or synthesis.

๐Ÿ“‘ Table of Contents


โœจ Key Features

Capability Implementation Advantage
Air-Gapped Privacy 100% Local (ONNX + SQLite + Embedded Qdrant + Ollama) Zero cloud API calls, zero tracking telemetry, zero data leaks.
Primary Data Integrity SQLite with WAL mode & cascading foreign keys Deterministic durability; vector store corruption never loses note data.
Zero-Waste Sync SHA-256 Content-addressable diffing Editing 1 paragraph only re-embeds that single chunk; unchanged chunks cost 0ms.
Lexical + Semantic Hybrid Qdrant ANN (bge-small-en-v1.5) + BM25Okapi (rank-bm25) Captures conceptual meaning alongside exact variable names, hashes, and IDs.
Reciprocal Rank Fusion RRF ($k=60$) score normalization Merges disparate score scales fairly without manual heuristic weighting.
Stage-2 Re-ranking FlashRank ONNX cross-encoder (bge-reranker-base) Cross-attention scoring elevates context precision while remaining CPU-efficient.
Small-to-Big Assembly Granular chunks (~350t) linked to parent Section IDs High search granularity without fragmented, out-of-context retrieval chunks.
Protocol-Driven Core Pure typing.Protocol interfaces with Pydantic v2 Fully decoupled layers; passes mypy --strict with zero warnings.

๐Ÿ›๏ธ Architectural Blueprint

End-to-End Workflow Diagram

flowchart TD
    subgraph INGESTION["1. INGESTION & PARSING"]
        A["Local Markdown Files (*.md)"] --> B["LocalFileReader"]
        B -->|YAML Frontmatter & H1 Detection| C["Canonical Document Model"]
        C --> D["MarkdownChunker"]
        D -->|AST Heading Traversal| E["Parent Sections (H1-H6 Context)"]
        D -->|Sliding Window Tokenizer| F["Granular Chunks (250-400 tokens)"]
        F -->|SHA-256 Content Hashing| G["Content-Addressed Chunks"]
    end

    subgraph CANONICAL_STORAGE["2. CANONICAL ACID STORAGE"]
        C -->|INSERT Document| H[("SQLite Primary Database\n(data/canonical.db)\nPRAGMA journal_mode=WAL")]
        E -->|INSERT Sections| H
        G -->|INSERT Chunks| H
    end

    subgraph SYNC_ENGINE["3. DIFFERENTIAL RECONCILIATION ENGINE"]
        H -->|get_all_chunks & get_active_chunk_hashes| I["IndexSyncEngine"]
        J[("Embedded Qdrant Index\n(data/qdrant)")] -->|get_all_point_ids| I
        I -->|Hash & ID Diff| K{"Reconciliation Delta"}
        K -->|Unchanged Hashes| L["Bypass Inference (0ms compute)"]
        K -->|Orphaned IDs| M["Purge Points from Qdrant & BM25"]
        K -->|New Chunk Text| N["FastEmbed ONNX\n(bge-small-en-v1.5)"]
        N -->|Upsert Vectors & Light Payloads| J
        N -->|Index Tokens & Persist| O[("Sparse BM25 Index\n(data/bm25.json)")]
    end

    subgraph RETRIEVAL["4. DECOUPLED HYBRID RETRIEVAL (Sub-200ms, Zero LLM)"]
        P["User Query String"] --> Q["StrataHybridSearcher"]
        Q -->|embed_query| N
        N -->|384d Dense Vector| R["Qdrant ANN Cosine Search"]
        Q -->|Tokenize Query| S["BM25 Lexical Keyword Search"]
        R -->|Top Dense Candidates| T["Reciprocal Rank Fusion\n(RRF k=60)"]
        S -->|Top Sparse Candidates| T
        T -->|Top-30 Fused Candidates| U["Hydrate Chunks from SQLite"]
        U --> V["FlashRank Cross-Encoder\n(bge-reranker-base)"]
        V -->|Top-K Winner Chunks| W{"Context Expansion?"}
        W -->|--expand-section| X["Hydrate Parent Section from SQLite"]
        W -->|--window-size N| Y["Hydrate Adjacent Sibling Chunks from SQLite"]
        X & Y --> Z["SearchResult (Rich Terminal Panels)"]
    end

    subgraph SYNTHESIS["5. LOCAL SYNTHESIS (Optional Consumer)"]
        Z --> AA["Grounded Citation Prompt"]
        AA --> AB["Local Ollama Engine\n(qwen3.5:0.8b / qwen2.5:7b)"]
        AB --> AC["Strictly Grounded Answer + Citations [Doc, ยง Heading]"]
    end
Loading

Workflow Deep-Dive

1. Ingestion & AST Parsing

When running strata ingest <path>, the LocalFileReader discovers Markdown files while ignoring hidden paths and vendor folders (.git, .venv, node_modules).

  • Delimited YAML frontmatter (---) is extracted into typed metadata and normalized tags.
  • If a document has no frontmatter title, title extraction falls back to the file stem, and finally to the first Markdown # H1 heading.
  • A cryptographic SHA-256 hash is computed over the raw file content to track whole-document changes.

2. Hierarchical Small-to-Big Chunking

The MarkdownChunker uses tiktoken (cl100k_base) to split documents along semantic AST boundaries:

  1. Section Extraction: Traverses Markdown heading levels (# through ######), ignoring comments and code inside triple-backtick fences. It establishes breadcrumb paths (e.g., Architecture > Storage > SQLite) and preserves each parent section's complete text as parent context.
  2. Granular Chunk Slicing: Sections exceeding the target chunk size (~350 tokens) are split into sliding windows with configurable overlap (~50 tokens). Short sections meeting the minimum threshold (~50 tokens) are preserved intact.
  3. Deterministic Content Hashing: Every chunk is assigned a deterministic hash: $$\text{chunk_hash} = \text{SHA-256}(\text{heading_path} : \text{chunk_text})$$
  4. Relational ACID Write: The Document, its Section records, and its Chunk records are written to SQLite within an atomic transaction. If the document previously existed, older records are deleted via ON DELETE CASCADE.

3. Differential Synchronization Engine

The IndexSyncEngine guarantees synchronization between SQLite and the derived search indices:

  • Identification: Compares canonical chunk IDs in SQLite against indexed vector IDs in embedded Qdrant.
  • Orphan Purging: Chunks that were deleted from SQLite are batch-removed from both Qdrant and the BM25 index.
  • Selective Embedding: Only newly created or altered chunks are sent to FastEmbedder (BAAI/bge-small-en-v1.5).
  • Lightweight Payloads: Qdrant payloads store strictly metadata (chunk_id, document_id, heading_path, tags, parent_section_id). Full text bodies are never duplicated inside Qdrant, preventing memory bloat.
  • Sparse Inverted Index: Chunks are tokenized and registered in BM25Index, and saved to data/bm25.json.

4. Decoupled Two-Stage Hybrid Retrieval

When querying with strata search, StrataHybridSearcher executes two search legs in parallel without loading any LLM:

  1. Dense Semantic ANN: Encodes the query into a 384-dimensional dense vector and executes cosine similarity search in embedded Qdrant, supporting metadata pre-filtering by --tag.
  2. Sparse Lexical BM25: Tokenizes the query into keywords and evaluates BM25Okapi scores over term frequencies and inverse document frequencies.
  3. Reciprocal Rank Fusion (RRF): Merges the disparate score spaces into a single ranking: $$\text{RRF}(d) = \sum_{m \in {\text{dense}, \text{sparse}}} \frac{w_m}{k + \text{rank}_m(d)}$$ where $k = 60$ (smoothing constant) and $w_m$ are retrieval leg weights.

5. Cross-Encoder Re-Ranking & Context Hydration

  1. FlashRank Cross-Attention: The top 30 fused candidate chunks are passed to FlashRankReranker (ms-marco-TinyBERT-L-2-v2). The cross-encoder attends across query and passage tokens simultaneously, assigning definitive relevance scores and selecting the top $K$ winners.
  2. Context Hydration:
    • Small-to-Big (--expand-section): Pulls the full parent Section text from SQLite using parent_section_id, providing complete conceptual context without sentence truncation.
    • Window Expansion (--window-size N): Fetches the $N$ chronologically preceding and succeeding sibling chunks from SQLite for adjacent narrative continuity.
  3. Output: Formats results with latency telemetry, scoring provenance, breadcrumbs, and matching passages using Rich terminal panels.

๐Ÿ“‚ Project Structure

Strata/
โ”œโ”€โ”€ src/
โ”‚   โ””โ”€โ”€ strata/
โ”‚       โ”œโ”€โ”€ __init__.py               # Package metadata and public exports
โ”‚       โ”œโ”€โ”€ core/                     # Pure domain layer (Zero I/O, zero external framework dependencies)
โ”‚       โ”‚   โ”œโ”€โ”€ config.py             # Pydantic Settings (env prefix STRATA_, frozen, typed)
โ”‚       โ”‚   โ”œโ”€โ”€ exceptions.py         # Domain error hierarchy (StrataError, SyncConflictError, etc.)
โ”‚       โ”‚   โ”œโ”€โ”€ hasher.py             # Content-addressable SHA-256 chunk & document hashing
โ”‚       โ”‚   โ”œโ”€โ”€ models.py             # Immutable Pydantic v2 models (Document, Section, Chunk, SearchResult)
โ”‚       โ”‚   โ””โ”€โ”€ protocols.py          # Structural subtyping interfaces (typing.Protocol, PEP 544)
โ”‚       โ”œโ”€โ”€ ingestion/                # Document discovery & structural AST splitting
โ”‚       โ”‚   โ”œโ”€โ”€ reader.py             # LocalFileReader (Frontmatter parser, H1 extraction, dir scanner)
โ”‚       โ”‚   โ””โ”€โ”€ chunker.py            # MarkdownChunker (Small-to-Big AST parser & tiktoken tokenizer)
โ”‚       โ”œโ”€โ”€ storage/                  # Canonical persistence layer
โ”‚       โ”‚   โ””โ”€โ”€ sqlite_store.py       # SQLiteStore (ACID primary store, WAL mode, cascading deletions)
โ”‚       โ”œโ”€โ”€ indexing/                 # Derived search representations & synchronization
โ”‚       โ”‚   โ”œโ”€โ”€ embedder.py           # FastEmbedder (Local BAAI/bge-small-en-v1.5 ONNX runtime)
โ”‚       โ”‚   โ”œโ”€โ”€ qdrant_store.py       # QdrantVectorStore (Embedded local Qdrant client adapter)
โ”‚       โ”‚   โ”œโ”€โ”€ bm25_index.py         # BM25Index (Inverted lexical index with rank-bm25 & persistence)
โ”‚       โ”‚   โ””โ”€โ”€ sync_engine.py        # IndexSyncEngine (Differential reconciliation & hash invalidation)
โ”‚       โ”œโ”€โ”€ retrieval/                # Decoupled search & fusion algorithms
โ”‚       โ”‚   โ”œโ”€โ”€ rrf.py                # Reciprocal Rank Fusion implementation
โ”‚       โ”‚   โ”œโ”€โ”€ reranker.py           # FlashRankReranker (Lightweight CPU ONNX cross-encoder)
โ”‚       โ”‚   โ””โ”€โ”€ hybrid_search.py      # StrataHybridSearcher (Two-stage hybrid coordinator)
โ”‚       โ””โ”€โ”€ cli/                      # Presentation layer (Typer + Rich)
โ”‚           โ”œโ”€โ”€ dependencies.py       # Dependency container & component factory
โ”‚           โ”œโ”€โ”€ views.py              # Rich terminal renderers (tables, panels, status cards)
โ”‚           โ””โ”€โ”€ main.py               # Typer CLI application commands (ingest, sync, search, status)
โ”œโ”€โ”€ tests/
โ”‚   โ”œโ”€โ”€ conftest.py                   # Pytest fixtures, in-memory databases, temporary directories
โ”‚   โ””โ”€โ”€ unit/                         # 279 comprehensive unit & integration tests
โ”‚       โ”œโ”€โ”€ test_bm25_index.py
โ”‚       โ”œโ”€โ”€ test_chunker.py
โ”‚       โ”œโ”€โ”€ test_cli.py
โ”‚       โ”œโ”€โ”€ test_core.py
โ”‚       โ”œโ”€โ”€ test_embedder.py
โ”‚       โ”œโ”€โ”€ test_hybrid_search.py
โ”‚       โ”œโ”€โ”€ test_qdrant_store.py
โ”‚       โ”œโ”€โ”€ test_reader.py
โ”‚       โ”œโ”€โ”€ test_reranker.py
โ”‚       โ”œโ”€โ”€ test_rrf.py
โ”‚       โ”œโ”€โ”€ test_sqlite_store.py
โ”‚       โ””โ”€โ”€ test_sync_engine.py
โ”œโ”€โ”€ data/                             # Default local storage directory (gitignored)
โ”‚   โ”œโ”€โ”€ canonical.db                  # Canonical SQLite database
โ”‚   โ”œโ”€โ”€ bm25.json                     # Persisted sparse index
โ”‚   โ””โ”€โ”€ qdrant/                       # Local embedded Qdrant vector storage
โ”œโ”€โ”€ pyproject.toml                    # Build config, dependencies, ruff & mypy configurations
โ”œโ”€โ”€ GEMINI.md                         # Architecture constraints, invariants, & coding conventions
โ””โ”€โ”€ README.md                         # Project documentation

๐Ÿš€ Local Setup & Getting Started

Prerequisites

  • Python 3.13+
  • uv (recommended extremely fast package and environment manager) or standard pip
  • (Optional) Ollama installed and running locally if you want to use local generative synthesis (qwen3.5:0.8b or qwen2.5:7b-instruct).

Installation with uv

# 1. Clone the repository
git clone https://github.com/your-username/Strata.git
cd Strata

# 2. Create and activate a virtual environment
uv venv
# On Linux/macOS:
source .venv/bin/activate
# On Windows:
.venv\Scripts\activate

# 3. Install Strata with development and evaluation dependencies
uv pip install -e ".[dev,eval]"

Alternatively, using standard pip:

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e ".[dev,eval]"

Local LLM Configuration (Optional)

If planning to run local LLM generation or offline RAGAS benchmarks:

# Pull the recommended default small synthesis model
ollama pull qwen3.5:0.8b

# Verify your local Ollama daemon is running
curl http://localhost:11434/api/tags

๐Ÿ’ป CLI Usage Guide

Strata provides an intuitive terminal interface powered by Typer and Rich.

1. Document Ingestion (strata ingest / strata add)

Ingest individual Markdown files or entire directories into SQLite:

# Ingest a single Markdown note with custom tags
strata ingest ./notes/system-architecture.md --tag architecture --tag backend

# Ingest an entire directory of notes recursively
strata ingest ./notes/ --tag personal

# 'add' is supported as a direct alias for 'ingest'
strata add ./notes/raft-consensus.md -t distributed-systems

Output:

Ingestion Summary
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Entity          โ”ƒ Count โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ Total Documents โ”‚     1 โ”‚
โ”‚ Total Sections  โ”‚     4 โ”‚
โ”‚ Total Chunks    โ”‚    12 โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
Ingestion complete: created 1 documents, 4 sections, and 12 chunks.

2. Differential Synchronization (strata sync)

Reconciles the derived Qdrant vector index and BM25 sparse index against SQLite:

# Run incremental synchronization
strata sync

# Override the embedding batch size if desired
strata sync --batch-size 64

Output:

Index Synchronization Summary
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Operation              โ”ƒ Record Count โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ Added Records          โ”‚            2 โ”‚
โ”‚ Updated Records        โ”‚            0 โ”‚
โ”‚ Deleted Records        โ”‚            1 โ”‚
โ”‚ Unchanged Records      โ”‚           10 โ”‚
โ”‚ Total Active Chunks    โ”‚           12 โ”‚
โ”‚ Total Documents        โ”‚            1 โ”‚
โ”‚ Elapsed Execution Time โ”‚    142.30 ms โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Note: Running strata sync again without modifying notes yields 0 Added, 12 Unchanged, executing in milliseconds without any tensor embedding computation.

3. Sub-200ms Hybrid Search (strata search)

Execute pure hybrid retrieval (Dense + Sparse + RRF + Cross-Encoder) with zero LLM overhead:

# Fast hybrid search (returns top 3 results)
strata search "How does leader election handle network partitions?" --limit 3

# Filter search candidates by tags
strata search "consensus invariants" --tag distributed-systems

# Enable Small-to-Big context expansion (hydrates full parent section from SQLite)
strata search "heartbeat timeout" --expand-section

# Expand window to include 1 adjacent sibling chunk before and after
strata search "Raft safety rules" --window-size 1

Sample Search Panel Output:

Found 1 matching result(s) for query: "How does leader election handle network partitions?"

โ•ญโ”€ Result #1 (Latency: 64.12ms) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ Document: Raft Consensus Notes                                                             โ”‚
โ”‚ Heading Path: Architecture > Raft > Leader Election                                        โ”‚
โ”‚ Rerank Score: 0.9412                                                                       โ”‚
โ”‚                                                                                            โ”‚
โ”‚ โ•ญโ”€ Matching Chunk โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚
โ”‚ โ”‚ If a follower receives no communication over election timeout, it assumes there is no  โ”‚ โ”‚
โ”‚ โ”‚ viable leader and transitions to candidate state, incrementing its current term...     โ”‚ โ”‚
โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ”‚
โ”‚                                                                                            โ”‚
โ”‚ โ•ญโ”€ Parent Section Context (Leader Election) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚
โ”‚ โ”‚ ## Leader Election                                                                     โ”‚ โ”‚
โ”‚ โ”‚ Raft uses a heartbeat mechanism to trigger leader election. Leaders send periodic      โ”‚ โ”‚
โ”‚ โ”‚ heartbeats (AppendEntries RPCs with no log entries) to maintain authority...           โ”‚ โ”‚
โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

4. System Telemetry & Status (strata status)

Inspect the health, record counts, and storage footprints across SQLite, Qdrant, and BM25:

strata status

Output:

Strata System Status & Telemetry
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Subsystem             โ”ƒ Metric                  โ”ƒ               Value โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ SQLite (Canonical)    โ”‚ Total Documents         โ”‚                   8 โ”‚
โ”‚ SQLite (Canonical)    โ”‚ Total Sections          โ”‚                  34 โ”‚
โ”‚ SQLite (Canonical)    โ”‚ Total Chunks            โ”‚                 112 โ”‚
โ”‚ Qdrant (Dense Vector) โ”‚ Active Vector Points    โ”‚                 112 โ”‚
โ”‚ BM25 (Sparse Index)   โ”‚ Index File Status       โ”‚ Active (persisted)  โ”‚
โ”‚ BM25 (Sparse Index)   โ”‚ Index Byte Size on Disk โ”‚        54,210 bytes โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

5. Canonical Database Visualization

To visualize the hierarchical structure of the canonical database, use the the following commands:

uv run --with datasette datasette data/canonical.db

Then open http://localhost:8001 in your browser.


โš™๏ธ Configuration Reference

All settings can be customized via environment variables prefixed with STRATA_ or via a root .env file (managed by Settings):

Variable Default Value Description
STRATA_DATA_DIR ./data Root directory for local data persistence
STRATA_DB_PATH ./data/canonical.db Path to canonical SQLite ACID database
STRATA_QDRANT_PATH ./data/qdrant Directory for embedded Qdrant vector index
STRATA_BM25_INDEX_PATH ./data/bm25.json Filesystem path for persisted BM25 sparse index
STRATA_EMBEDDING_MODEL BAAI/bge-small-en-v1.5 Dense embedding model (FastEmbed ONNX runtime)
STRATA_EMBEDDING_DIMENSION 384 Dimensionality of dense embedding vectors
STRATA_RERANKER_MODEL ms-marco-TinyBERT-L-2-v2 Cross-encoder re-ranking model (FlashRank ONNX)
STRATA_CHUNK_SIZE 350 Target search chunk length in tokens
STRATA_CHUNK_OVERLAP 50 Token overlap between adjacent chunks
STRATA_RRF_K 60 Reciprocal Rank Fusion smoothing constant
STRATA_FUSED_TOP_K 30 Number of fused candidates routed to cross-encoder
STRATA_FINAL_TOP_K 5 Default number of final re-ranked results returned
STRATA_OLLAMA_BASE_URL http://localhost:11434 Local Ollama service endpoint
STRATA_LLM_MODEL qwen3.5:0.8b Default local LLM tag for synthesis

๐Ÿงช Quality Assurance & Verification

Strata enforces strict software engineering standards with automated linting, formatting, static analysis, and test suites:

# 1. Run full test suite (279 passing unit & integration tests)
uv run pytest -v

# 2. Strict static type analysis (zero mypy warnings)
uv run mypy --strict src/

# 3. Fast code style, linting, and formatting checks
uv run ruff check src/ tests/
uv run ruff format --check src/ tests/

๐Ÿ“„ License

This project is licensed under the MIT License. See the LICENSE file for complete details.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages