This is the full developer documentation for Persia DB # Persia DB > An embedded, single-file, cloud-native search database, written in Rust. Pre-alpha. Think SQLite for search. Persia DB is an embedded, single-file, cloud-native search database, written in Rust. Think “SQLite for search”: open a file, index documents, and get ranked results that, by default, never come back empty. The same segment bytes can live in a local file or in object storage (S3, GCS, or Azure Blob) and can be served by stateless query nodes on Kubernetes. Pre-alpha Nothing works end to end yet, and the on-disk format is unstable until v1.0. There is no compatibility promise ([SPEC §15](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#15-versioning-and-compatibility)). The live counter is [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md): 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. The [roadmap](https://tomiok.github.io/persiaDB/roadmap/) page breaks that snapshot down by milestone. []()Pause ``` ╔═ layers ═════════════════════════════════════════════════════════╗ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-server, Rust SDK, Go SDK ┃ ║░ ║ ┃ planned ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia · Db and Collection ┃ ║░ ║ ┃ stub · no methods yet ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ engine ┃ ┃ blob ┃ ┃ analysis ┃ ║░ ║ ┃ WAL and query ┃ ┃ chunks, dedup ┃ ┃ analyzers ┃ ║░ ║ ┃ in progress ┃ ┃ not started ┃ ┃ done ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-storage ┃ ║░ ║ ┃ memory and local file ┃ ║░ ║ ┃ s3, gcs, azure planned ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-format ┃ ║░ ║ ┃ encodings, checksums, codecs ┃ ║░ ║ ┃ done ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ╚══════════════════════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ``` Layers from [SPEC §3](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#3-architecture-overview). Status from the 2026-11-05 PROGRESS.md snapshot. Format, local storage, schema, and text analysis are done (M1–M4). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. The blob store is not started (M9; its dependencies are done). The embedded API has not been started (M10), and the persia crate exports no Db or Collection. The server is not started (M11). Network SDKs are not started (M14). S3, GCS, and Azure are not started (M12). Arrows point down: each layer is used by the one above it. [Getting started](https://tomiok.github.io/persiaDB/guide/getting-started/)[Read the guide](https://tomiok.github.io/persiaDB/guide/overview/)[Features](https://tomiok.github.io/persiaDB/features/)[How it works](https://tomiok.github.io/persiaDB/architecture/)[Roadmap](https://tomiok.github.io/persiaDB/roadmap/)[GitHub](https://github.com/tomiok/persiaDB) ## Waitlist [Section titled “Waitlist”](#waitlist) Nothing here is ready to run. Leave an email if you want a note when that changes. What the form stores is described on the [waitlist](https://tomiok.github.io/persiaDB/waitlist/) page. The waitlist is not open yet. The public key or the relay URL is not configured. EmailName (optional)Company (optional)Use case (optional) Leave this empty Join the waitlist JavaScript is required. Encryption runs in the browser, so this form does not post the address anywhere without it. ## Why this exists [Section titled “Why this exists”](#why-this-exists) | Property | What it means | Status | | ----------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Zero friction | `Db::open("app.persia")` just works. No daemon and no config file for the embedded library. | Planned (M10) | | Own engine | Index format, analysis, BM25F scoring, and query execution are written in this repository. | Format codecs, text analysis, and a segment writer and reader exist. The write path is in progress (M6, 49/58): WAL, commit, and segment merging. BM25F scoring and a collection search are in the tree (CollectionViews::search). There is no search call on the database handle. | | Never zero results | A query that matches nothing is relaxed step by step, and the response says what changed. | Not started (M8). [How the cascade is designed](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). | | Cloud-native | Immutable segments. The same bytes locally and in object storage. Stateless query nodes. | Segment bytes can be written (M5 done, 48/48). The write path is in progress (M6, 49/58): WAL, commit, and segment merging. Object storage has not been started (M12). Query nodes have not been started (M13). | | Correctness, then speed | Checksums, crash-safe commits, and fuzzed parsers come before performance claims. | Checksums, fuzzed parsers, the container commit, and segment read/write exist. The write path is in progress (M6, 49/58): WAL, commit, and segment merging. A collection snapshot can be searched as library code. There is no search call on the database handle. | The design contract is [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md) (draft v0.1). When this site and the SPEC disagree, the SPEC wins. ## Where to go next [Section titled “Where to go next”](#where-to-go-next) * [Getting started](https://tomiok.github.io/persiaDB/guide/getting-started/) separates the codecs you can call from the database API, which is still illustrative. * [What is an embedded search database?](https://tomiok.github.io/persiaDB/guide/embedded-search/) explains in-process, single-file full-text search, and which parts of that design exist in the repository. * [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/) explains query relaxation, fuzzy matching, BM25F, and filters: what is designed, and what the engine implements. * [Comparison](https://tomiok.github.io/persiaDB/guide/comparison/) places the design next to SQLite FTS5, Tantivy, and Meilisearch. Persia DB has no `persia::Db` search API yet (M10); the engine library code can search. * [Guide overview](https://tomiok.github.io/persiaDB/guide/overview/) is the full narrative, split from the repository guide. * [For AI agents](https://tomiok.github.io/persiaDB/guide/for-agents/) is the short list of constraints. [llms.txt](https://tomiok.github.io/persiaDB/llms.txt) indexes the pages. * [About](https://tomiok.github.io/persiaDB/about/) covers license, conduct, and how to contribute. # Getting started with PersiaDB > Getting started with PersiaDB, a pre-alpha Rust search library: what you can call today, and the planned Db API. Getting started with PersiaDB means separating this Rust search library’s design from the code you can call. The design is embedded full-text search in Rust: a library in your process, one file, ranked results, and no search daemon. Persia DB is pre-alpha. The public persia crate has no Db::open. persia-engine can put, delete, and commit a document. persia-engine runs a ranked search over a collection snapshot: BM25F, filters, sorts, facets, and pages. The first release a caller is supposed to use is v0.1 (M10). That milestone is blocked on M6 (in progress), M7 (in progress), M8 (not started), and M9 (not started). What you can call today is library code. persia-engine creates a database file, writes documents, and searches a collection snapshot; the [README quickstart](https://github.com/tomiok/persiaDB#index-documents-and-search-them) runs those calls. That is not `persia::Db::open`. [What is an embedded search database?](https://tomiok.github.io/persiaDB/guide/embedded-search/) defines that in-process shape. The [comparison](https://tomiok.github.io/persiaDB/guide/comparison/) places the design next to SQLite FTS5, Tantivy, and Meilisearch, and it is not a benchmark. The [FAQ](https://tomiok.github.io/persiaDB/guide/faq/) has the short answers. When this page and the [SPEC](https://github.com/tomiok/persiaDB/blob/main/SPEC.md) disagree, the SPEC wins. Live status is [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md): 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. The same snapshot, by milestone, is the [roadmap](https://tomiok.github.io/persiaDB/roadmap/). ## What this Rust search library implements today [Section titled “What this Rust search library implements today”](#what-this-rust-search-library-implements-today) The crates are not published (`publish = false` in the workspace). There is no crates.io release and no `cargo add persia`. `persia`, the embedded API crate, exports no types. Its source is the module docs in [lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia/src/lib.rs). There is no `Db` and no `Collection`. `persia-server` is a placeholder binary: it prints “not implemented (M11)” and exits with status 1. No gRPC or HTTP server runs. `persia-cli` lists and removes temporary and quarantine files next to a container or a local store (`persia container files`, `persia container sweep`, `persia store files`, `persia store sweep`). It can inspect and dump one segment file (`persia inspect segment`, `persia dump segment`, with `--verify` on inspect), and check a whole database read-only (`persia verify`). It does not compact, import, or search. `persia-format` encodes and decodes on-disk bytes. It does not open a file as a database. A git dependency, for code that wants those codecs: ```toml [dependencies] persia-format = { git = "https://github.com/tomiok/persiaDB" } ``` This round-trips a container header (SPEC §4.2). It is the doctest on [`Header`](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/header.rs). It does not create `app.persia` and it does not search: ```rust use persia_format::header::{Access, HEADER_LEN, Header}; use persia_format::{Reader, Writer}; let header = Header::new([7; 16]); let mut w = Writer::new(); header.encode(&mut w)?; assert_eq!(w.len(), HEADER_LEN); let decoded = Header::decode(&mut Reader::new(w.as_slice()))?; assert_eq!(decoded, header); assert_eq!(decoded.access(), Access::ReadWrite); ``` Also implemented in that crate, and only as byte codecs: checksums, integer blocks, zstd block compression, block framing, the two superblock slots, frames (including the streaming encoder), the memory-map reader, and `BlobId` / `BlobRef`. The public module list is [lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/lib.rs). The [code](https://tomiok.github.io/persiaDB/guide/code/) page quotes exports from that file. persia-format implements the on-disk codecs (M1 done, 53/53). persia-storage implements the Storage trait, memory and local-file backends, and the single-file container (M2 done, 46/46). S3, GCS, and Azure have not been started (M12). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. persia-analysis implements tokenizers and the built-in analyzers de, en, es, fr, it, keyword, pt, standard, and whitespace (M4 done, 29/29). It does not search. persia-blob exports no API (M9 not started; its dependencies are done). The public persia crate exports no Db and no Collection. The crates are unpublished (publish = false, version 0.0.0). Those crates are in the same git repository and are unpublished the same way. Depending on them does not give you `Db::open`. ## Embedded full-text search in Rust [Section titled “Embedded full-text search in Rust”](#embedded-full-text-search-in-rust) **Illustrative public API.** The block below is the SPEC sketch of `persia::Db` (SPEC §6.2, §7.1, §7.6, §8.1, and §9.6). It is not a stability promise. persia_engine::db::Database has open, create_collection, put, put_batch, delete, and commit. CollectionSnapshot::get fetches one document by id. CollectionViews::search runs a ranked search over a collection snapshot. There is no persia::Db, no search on the database handle, no put_struct, no put_blob, no get_blob, and no OpenOptions. Those SPEC names are not the public embedded API. The first release a caller is supposed to use is v0.1 (M10). That milestone is blocked on M6 (in progress), M7 (in progress), M8 (not started), and M9 (not started). The shape a caller is meant to use: ```rust // ILLUSTRATIVE — these methods are not in the persia crate. let db = persia::Db::open("app.persia")?; // Schema is declared up front (SPEC §6.2). Adding a field later is allowed. // Changing a field's type is not. let products = db.create_collection(schema)?; products.put(doc)?; products.commit()?; // A typo still returns hits. The response says what changed (SPEC §7.6). let results = products.search("runing")?; let relaxation = &results.relaxation; ``` The collection schema in SPEC §6.2, as JSON. `CollectionSchema::from_json` in `persia-engine` parses a schema. The block below is the SPEC sketch, and it does not open a database: ```json { "name": "products", "fields": { "title": { "type": "text", "analyzer": "en", "boost": 3.0 }, "body": { "type": "text", "analyzer": "en" }, "brand": { "type": "keyword", "facet": true }, "price": { "type": "f64" }, "sales": { "type": "i64" } }, "ranking": { "static_rank": { "field": "sales", "weight": 0.1, "fn": "log1p" } }, "relaxation": { "enabled": true, "min_results": 1, "levels": "default", "fallback": "static_rank" } } ``` Field types in that schema are `text`, `keyword`, `i64`, `f64`, `bool`, `datetime`, and `stored` (SPEC §6.1). Every document also has a string `id` (at most 512 bytes, unique in the collection). Documents are submitted as JSON, or as a `serde` struct via a planned `put_struct`, and stored as CBOR (SPEC §6.4). `put` is an upsert by id. `delete` writes a tombstone. `commit` publishes a manifest. A `put` is specified to be visible to a later search on that database before commit (read-your-writes, SPEC §8.2–§8.4). One writer; a second writer fails immediately. Handles are specified as `Send` and `Sync`. Search is specified to accept a text query or a structured query (SPEC §7.1): fields and boosts, filters, sort, `limit` / `offset`, facets, and `relax`. Scoring is BM25F plus an optional static rank (SPEC §7.2). ### Relaxation [Section titled “Relaxation”](#relaxation) Not started (M8). On by default in the design. The collection search is library code; this cascade is not, so a search that matches nothing returns no hits. If a query has fewer than `min_results` hits (default 1), the engine walks a cascade and stops at the first level that has enough hits (SPEC §7.6). The longer page is [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). | Level | Name | What changes | | ----- | ------------ | ---------------------------------------------------- | | L0 | `exact` | Every analyzed term is required | | L1 | `synonyms` | Collection synonyms, OR within a group | | L2 | `drop_terms` | Require fewer terms, dropping low-IDF terms first | | L3 | `prefix` | Prefix-expand terms of length at least 3 | | L4 | `fuzzy` | Edit distance 1 or 2, and report the corrected terms | | L5 | `fallback` | `static_rank`, or `none` | Filters are not relaxed unless the caller sets `relax.filters` to `"last_resort"`. `relax: false` on a query turns the cascade off. The response is specified to carry what happened. This object is from SPEC §7.6. No function returns it today: ```json { "level": "fuzzy", "applied": ["drop_terms:75%", "fuzzy"], "original": "zapatilas rojs", "effective": "zapatillas rojas", "corrected": { "zapatilas": "zapatillas", "rojs": "rojas" } } ``` ### Command line [Section titled “Command line”](#command-line) **Partly in the tree.** `persia inspect segment` and `persia dump segment` read one segment file, and `--verify` checks it. `persia verify` checks a whole database. Still planned (ROADMAP 10.4): `compact`, `stats`, JSONL import, a one-shot search, and a small bench. The binary does none of those. ## Common questions [Section titled “Common questions”](#common-questions) The [FAQ](https://tomiok.github.io/persiaDB/guide/faq/) answers the same kind of question for status, how this differs from SQLite FTS5, Tantivy, and Meilisearch, the file format, crash safety, SDKs, and the license. ### Can I add Persia to a Rust project today? [Section titled “Can I add Persia to a Rust project today?”](#can-i-add-persia-to-a-rust-project-today) Not as a database. The public persia crate exports no Db and no Collection. A search runs only through persia-engine (CollectionViews::search). The crates are unpublished (publish = false, version 0.0.0). You can depend on `persia-format` from git, as shown above, and on `persia-storage`, `persia-engine`, and `persia-analysis`. persia_engine::db::Database has open, create_collection, put, put_batch, delete, and commit. CollectionSnapshot::get fetches one document by id. CollectionViews::search runs a ranked search over a collection snapshot. There is no persia::Db, no search on the database handle, no put_struct, no put_blob, no get_blob, and no OpenOptions. Those SPEC names are not the public embedded API. ### How do I open a database, index, and search? [Section titled “How do I open a database, index, and search?”](#how-do-i-open-a-database-index-and-search) You don’t, yet, through `persia::Db::open`. Those calls are the illustrative block on this page. persia_engine::db::Database has open, create_collection, put, put_batch, delete, and commit. CollectionSnapshot::get fetches one document by id. CollectionViews::search runs a ranked search over a collection snapshot. There is no persia::Db, no search on the database handle, no put_struct, no put_blob, no get_blob, and no OpenOptions. Those SPEC names are not the public embedded API. Below that API, `persia_engine::db::Database` opens a file, and the [README quickstart](https://github.com/tomiok/persiaDB#index-documents-and-search-them) searches it. The first release a caller is supposed to use is v0.1 (M10). That milestone is blocked on M6 (in progress), M7 (in progress), M8 (not started), and M9 (not started). [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) is the checklist. ### How do I read which relaxation was applied? [Section titled “How do I read which relaxation was applied?”](#how-do-i-read-which-relaxation-was-applied) On the intended API, from the search response field `relaxation` (SPEC §7.6): `level`, `applied`, `original`, `effective`, and `corrected`. Stricter levels rank above looser ones. Nothing in the repository computes this. See [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). ### Is the on-disk format stable? [Section titled “Is the on-disk format stable?”](#is-the-on-disk-format-stable) No. Format version 1.0 is a draft until the v1.0 release (SPEC §15). There is no compatibility promise. A change to the bytes requires a SPEC edit, a version or feature-flag bump, an ADR, and a golden fixture. ### Is there a server, a cloud backend, or a client SDK? [Section titled “Is there a server, a cloud backend, or a client SDK?”](#is-there-a-server-a-cloud-backend-or-a-client-sdk) Planned, not started. gRPC and HTTP are M11. S3, GCS, and Azure Blob are M12. Kubernetes sharding is M13. Rust and Go network clients are M14. Embedded Go and TypeScript bindings, which would open a file in-process, are a separate design ([Embedded SDKs](https://tomiok.github.io/persiaDB/guide/sdks/)); they are not implemented either. Embedded mode, when it exists, has no authentication and no config file (SPEC §14). ### Why is a query supposed to avoid empty results? [Section titled “Why is a query supposed to avoid empty results?”](#why-is-a-query-supposed-to-avoid-empty-results) That is a product rule (SPEC §7.6), not an optional plugin. A query that matches nothing is relaxed step by step, and the response says what changed. Not started (M8). On by default in the design. The collection search it would wrap is library code. [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/) separates the two. ### Where is the source of truth? [Section titled “Where is the source of truth?”](#where-is-the-source-of-truth) [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md) for behavior. [ROADMAP.md](https://github.com/tomiok/persiaDB/blob/main/ROADMAP.md) for planned work. [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) for what is done. This site is a reading of those files. [For AI agents](https://tomiok.github.io/persiaDB/guide/for-agents/) is the short version of the same constraints. # Persia DB FAQ > What Persia DB is, whether it is production ready, how it differs from SQLite FTS5, Tantivy, and Meilisearch, plus crash safety and the Apache-2.0 license. Persia DB is pre-alpha. These answers describe the repository as of [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md): 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. That file is the live counter. [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md) wins if a page disagrees with it. Planned work is marked as planned. Nothing here is a benchmark. ## What is Persia DB? [Section titled “What is Persia DB?”](#what-is-persia-db) Persia DB is an embedded, single-file, cloud-native search database written in Rust. The design is “SQLite for search”: open a file, index documents, and get ranked results that, by default, never come back empty. The same segment bytes are specified to live in a local file or in object storage (S3, GCS, or Azure Blob), and to be served by stateless query nodes. It is pre-alpha. The design contract is [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md). When this site and the SPEC disagree, the SPEC wins. What the repository contains today is library code, not the public persia::Db handle you can open. persia-format implements the on-disk codecs (M1 done, 53/53). persia-storage implements the Storage trait, memory and local-file backends, and the single-file container (M2 done, 46/46). S3, GCS, and Azure have not been started (M12). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. persia-analysis implements tokenizers and the built-in analyzers de, en, es, fr, it, keyword, pt, standard, and whitespace (M4 done, 29/29). It does not search. persia-blob exports no API (M9 not started; its dependencies are done). The public persia crate exports no Db and no Collection. The crates are unpublished (publish = false, version 0.0.0). Live status is [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md). What an embedded search database is, and how embedded full-text search in Rust fits this design, is [What is an embedded search database?](https://tomiok.github.io/persiaDB/guide/embedded-search/). ## Is Persia DB production ready? [Section titled “Is Persia DB production ready?”](#is-persia-db-production-ready) No. Persia DB is pre-alpha, and its public API (Db) does not exist yet. The crates are unpublished (`publish = false`, version 0.0.0). There is no crates.io release and no compatibility promise. The on-disk format stays unstable until v1.0 (roadmap M15, SPEC §15). The first release a caller is supposed to use is v0.1 (M10). That milestone is blocked on M6 (in progress), M7 (in progress), M8 (not started), and M9 (not started). [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) is the live counter: 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. A number on this page is that snapshot. For a search feature in production today, use an existing engine. The [comparison page](https://tomiok.github.io/persiaDB/guide/comparison/) says which one fits which job, and it is not a benchmark. ## How is Persia DB different from SQLite FTS5, Tantivy, and Meilisearch? [Section titled “How is Persia DB different from SQLite FTS5, Tantivy, and Meilisearch?”](#how-is-persia-db-different-from-sqlite-fts5-tantivy-and-meilisearch) The longer page is [Persia DB vs SQLite FTS5, Tantivy, and Meilisearch](https://tomiok.github.io/persiaDB/guide/comparison/). It is not a benchmark, and this project does not publish speed numbers. SQLite FTS5 is a full-text module inside SQLite, a C library. You search with SQL, and the index lives in the SQLite database file. It is a released part of SQLite. Tantivy is a Rust full-text search library. Its index is a directory of segment files, and indexing and serving are your code. It is a released library. Persia DB does not use Tantivy. [ADR-0001](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0001-own-engine-and-allowed-dependencies.md) forbids depending on another search engine. Meilisearch is a standalone search server written in Rust. Clients use HTTP, and typo tolerance is on by default. It is a released server. The Persia DB server is planned (milestone M11) and is not written: `persia-server` is a placeholder binary that exits with “not implemented (M11)”. The Persia DB design is an embedded Rust library, one local .persia file, the same segment bytes in object storage, and query relaxation on by default. The query compiler, the filter DSL, and a BM25F scorer are library code. A ranked search over a collection snapshot is library code too. A search call on a database handle and the relaxation cascade are not. That split is [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). ## Is Persia DB embedded or a server? [Section titled “Is Persia DB embedded or a server?”](#is-persia-db-embedded-or-a-server) The design is an embedded library: in your process, open a file, with no daemon and no config file for the library (SPEC §14). An optional gRPC and HTTP server is planned (milestone M11) and is not implemented. See [Server, cloud, and Kubernetes](https://tomiok.github.io/persiaDB/guide/server/). [What is an embedded search database?](https://tomiok.github.io/persiaDB/guide/embedded-search/) is the longer explanation of that shape. What you can build against today is still a library, not that server. `persia-format` encodes and decodes on-disk bytes. `persia-storage` opens a single-file container and implements memory and local-file object stores. The `persia` command lists and removes temporary and quarantine files next to a container or a local store (`persia container files`, `persia container sweep`, `persia store files`, `persia store sweep`). It can also inspect and dump one segment file (`persia inspect segment`, `persia dump segment`, with `--verify` on inspect). It does not import documents or search. There is no `Db::open`. ## What languages and SDKs does Persia DB support? [Section titled “What languages and SDKs does Persia DB support?”](#what-languages-and-sdks-does-persia-db-support) The implementation language is Rust, and nothing is published for other languages. The embedded API crate `persia` exports no types, so there is no Rust database API to wrap yet. Embedded Go and TypeScript bindings are a design only ([ADR-0018](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0018-embedded-sdk-bindings.md), [Embedded SDKs](https://tomiok.github.io/persiaDB/guide/sdks/)). There is no Go module and no npm package. The directories `sdk/persia` and `sdk/node` do not exist. Rust and Go network clients (`persia-client`, `persia-go`) are milestone M14, after the server (M11) and sharding (M13). Those directories do not exist either. ## What file format does Persia DB use? [Section titled “What file format does Persia DB use?”](#what-file-format-does-persia-db-use) A local database is specified as two files (SPEC §4 and §9). The file `name.persia` is the container: a 12 KiB prefix (a 4 KiB header and two 4 KiB superblock slots) and then append-only frames. Integers are little-endian. Headers and frames use CRC32C. The file `name.persia.blobs` is a planned content-addressed pack for client bytes such as PDF or images. `BlobId` and `BlobRef` exist as types in `persia-format`. The blob store is milestone M9 and has not been started. `persia-blob` exports no API. Implemented today: the container header, superblock, frames (including the streaming encoder), checksums, integer codecs, zstd block compression, and block framing, in `persia-format`. Opening, appending, committing, recovering, and compacting a container file are implemented in `persia-storage`. persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. S3, GCS, and Azure Blob backends are milestone M12 and are not written. Format version 1.0 is a draft until the v1.0 release (SPEC §15). Details are on [File format and storage](https://tomiok.github.io/persiaDB/guide/file-format/). ## Is Persia DB crash-safe? [Section titled “Is Persia DB crash-safe?”](#is-persia-db-crash-safe) The single-file container commit is implemented and crash-tested. SPEC §4.5 orders the writes: data frames, then the manifest frame, then the inactive superblock slot, with an fdatasync at each step. A crash before the new superblock is durable leaves the previous commit. Opening the container runs recovery, keeps that commit, and truncates a torn tail. Create and compaction are specified to leave either no new file or a complete one. Tests inject faults at those steps and kill the process. Recovery must return the old state or the new state, and not a mix of the two. The container commit is done (M2, 46/46) and is crash-tested. The engine write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Database::open reloads the committed manifest, checks it against the superblock, and replays the WAL. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle. Checksums cover the bytes that exist, and fuzz targets cover the parsers that exist. ## What license is Persia DB under? [Section titled “What license is Persia DB under?”](#what-license-is-persia-db-under) Apache License, Version 2.0. The text is the [LICENSE](https://github.com/tomiok/persiaDB/blob/main/LICENSE) file in the repository root. Copyright and attribution are in [NOTICE](https://github.com/tomiok/persiaDB/blob/main/NOTICE): Copyright 2026 The Persia DB Authors. There is no second project license. The workspace Cargo.toml sets `license` to Apache-2.0. The [license page](https://tomiok.github.io/persiaDB/guide/license/) describes what that text says about embedding the library in an application. It is a description of the license, not legal advice. ## How do I get started with Persia DB? [Section titled “How do I get started with Persia DB?”](#how-do-i-get-started-with-persia-db) You cannot add Persia DB as a database yet. There is no `cargo add persia`. [Getting started](https://tomiok.github.io/persiaDB/guide/getting-started/) separates the library code you can call from the illustrative `Db::open` and `search` API, which is not in the crate. persia-format implements the on-disk codecs (M1 done, 53/53). persia-storage implements the Storage trait, memory and local-file backends, and the single-file container (M2 done, 46/46). S3, GCS, and Azure have not been started (M12). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. persia-analysis implements tokenizers and the built-in analyzers de, en, es, fr, it, keyword, pt, standard, and whitespace (M4 done, 29/29). It does not search. persia-blob exports no API (M9 not started; its dependencies are done). The public persia crate exports no Db and no Collection. The crates are unpublished (publish = false, version 0.0.0). persia_engine::db::Database has open, create_collection, put, put_batch, delete, and commit. CollectionSnapshot::get fetches one document by id. CollectionViews::search runs a ranked search over a collection snapshot. There is no persia::Db, no search on the database handle, no put_struct, no put_blob, no get_blob, and no OpenOptions. Those SPEC names are not the public embedded API. The first release a caller is supposed to use is v0.1 (M10). That milestone is blocked on M6 (in progress), M7 (in progress), M8 (not started), and M9 (not started). ## How can I contribute to Persia DB? [Section titled “How can I contribute to Persia DB?”](#how-can-i-contribute-to-persia-db) Read [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md) and [CLAUDE.md](https://github.com/tomiok/persiaDB/blob/main/CLAUDE.md), then pick one leaf from [ROADMAP.md](https://github.com/tomiok/persiaDB/blob/main/ROADMAP.md). [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) lists what is ready next. One leaf is one pull request. The [contributing page](https://tomiok.github.io/persiaDB/guide/contributing/) lists the checks a pull request has to pass, including format, clippy, tests, and the dependency and progress checks. By participating you agree to the [Code of Conduct](https://github.com/tomiok/persiaDB/blob/main/CODE_OF_CONDUCT.md). Report security issues privately, as described in [SECURITY.md](https://github.com/tomiok/persiaDB/blob/main/SECURITY.md). # For AI agents > Status, the API that exists, the API that is only specified, and the constraints for code that reads or writes this repository. Read this before generating a call to Persia or a patch against the repo. The design contract is [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md). If a page on this site disagrees with the SPEC, the SPEC wins. Machine-readable copies of these pages: [llms.txt](https://tomiok.github.io/persiaDB/llms.txt) and [llms-full.txt](https://tomiok.github.io/persiaDB/llms-full.txt). ## Status [Section titled “Status”](#status) Pre-alpha. 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. The live counter is [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md). Do not treat a number on an older page as newer than that file. Persia DB is pre-alpha. The public persia crate has no Db::open. persia-engine can put, delete, and commit a document. persia-engine runs a ranked search over a collection snapshot: BM25F, filters, sorts, facets, and pages. persia-format implements the on-disk codecs (M1 done, 53/53). persia-storage implements the Storage trait, memory and local-file backends, and the single-file container (M2 done, 46/46). S3, GCS, and Azure have not been started (M12). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. persia-analysis implements tokenizers and the built-in analyzers de, en, es, fr, it, keyword, pt, standard, and whitespace (M4 done, 29/29). It does not search. persia-blob exports no API (M9 not started; its dependencies are done). The public persia crate exports no Db and no Collection. The crates are unpublished (publish = false, version 0.0.0). The on-disk format is unstable until v1.0 (SPEC §15). | Question | Answer | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Usable database API? | No. The public persia crate exports no Db and no Collection. A search runs only through persia-engine (CollectionViews::search). The crates are unpublished (publish = false, version 0.0.0). | | Published on crates.io? | No. `publish = false`. | | CLI? | File sweep (`container` / `store` `files` and `sweep`), plus `inspect segment` and `dump segment` (`--verify` on inspect). Not compact, import, or search. | | Working library code? | persia-format implements the on-disk codecs (M1 done, 53/53). persia-storage implements the Storage trait, memory and local-file backends, and the single-file container (M2 done, 46/46). S3, GCS, and Azure have not been started (M12). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. persia-analysis implements tokenizers and the built-in analyzers de, en, es, fr, it, keyword, pt, standard, and whitespace (M4 done, 29/29). It does not search. persia-blob exports no API (M9 not started; its dependencies are done). The public persia crate exports no Db and no Collection. The crates are unpublished (publish = false, version 0.0.0). | | Next work? | The ready-next list in PROGRESS.md. | | First callable release? | The first release a caller is supposed to use is v0.1 (M10). That milestone is blocked on M6 (in progress), M7 (in progress), M8 (not started), and M9 (not started). | The walkthrough with the illustrative calls, and the FAQ, is [Getting started](https://tomiok.github.io/persiaDB/guide/getting-started/). ## Do not invent these calls [Section titled “Do not invent these calls”](#do-not-invent-these-calls) The following names appear in the SPEC and the README. persia_engine::db::Database has open, create_collection, put, put_batch, delete, and commit. CollectionSnapshot::get fetches one document by id. CollectionViews::search runs a ranked search over a collection snapshot. There is no persia::Db, no search on the database handle, no put_struct, no put_blob, no get_blob, and no OpenOptions. Those SPEC names are not the public embedded API. Do not emit `persia::Db::open` or `search` as working code, and do not describe them as the public API. A search result’s `relaxation` object (`level`, `applied`, `original`, `effective`, `corrected`) is specified in SPEC §7.6 and returned by nothing today. `QueryCompiler`, `Filter`, and `TermScorer` are in `persia-engine`. They compile text and score a term; `CollectionViews::search` runs the ranked search. [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/) is that split. There is no Go module and no npm package. `sdk/go` in the SPEC is a future gRPC client. The embedded Go and TypeScript design ([Embedded SDKs](https://tomiok.github.io/persiaDB/guide/sdks/), [ADR-0018](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0018-embedded-sdk-bindings.md)) is a document, not a library. Do not emit `import "github.com/tomiok/persiaDB/sdk/persia"` or `import { open } from "@persiadb/node"` as working code. Code that only needs the byte codecs can use `persia-format` (`Header`, `Reader`, `Writer`, `BlobId`, `BlobRef`, and the modules listed in [lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/lib.rs#L10-L40)). Quote those from the source. The [code](https://tomiok.github.io/persiaDB/guide/code/) page links the line ranges. ## Constraints on patches [Section titled “Constraints on patches”](#constraints-on-patches) [CLAUDE.md](https://github.com/tomiok/persiaDB/blob/main/CLAUDE.md) is the engineering ruleset. The short list: * The engine is written here. Do not add Tantivy, Lucene, SQLite, RocksDB, or another search or storage engine. * `unsafe` is allowed only in files listed in `xtask/src/repo_checks.rs` (`UNSAFE_ALLOWED`): `persia-format`’s mmap and the container’s mapped-read module. Each block needs a `SAFETY` comment and an ADR. ADR-0018 reserves one future module, the C exports of `persia-ffi`, which is not in the tree. * Bytes from disk, network, or callers return `Err`, never a panic. Corruption is `Error::Corrupt`. * A change to on-disk layout updates SPEC.md, bumps a version or feature flag, adds an ADR, and adds a golden test. * Segments are immutable once written. Data is written first, the commit record last, then fsync. * Dependencies point downward. The allowed list is the workspace `[workspace.dependencies]` and `cargo xtask check-deps`. * One roadmap leaf is one change. Do not mark a leaf done until its definition of done is met. ## How to read the docs [Section titled “How to read the docs”](#how-to-read-the-docs) Each guide page has a `description` in its frontmatter and stable headings. [llms.txt](https://tomiok.github.io/persiaDB/llms.txt) is the index. [llms-full.txt](https://tomiok.github.io/persiaDB/llms-full.txt) is the pages in Markdown. Prefer those, plus SPEC.md, over guessing from the crate names. `persia-engine`, `persia-storage`, and `persia-analysis` export the library code in the table above. `persia-blob` and `persia-proto` are module docs only. # Overview > What Persia DB is: an embedded, single-file search database written in Rust, what already works in pre-alpha, and why it is built this way. > **Status: pre-alpha.** Nothing works end to end yet, and the on-disk format is unstable until v1.0. There is no compatibility promise. Counts on the [milestones page](https://tomiok.github.io/persiaDB/guide/roadmap/) are read from [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) when the site is built: 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md) is the design contract (draft v0.1, format version 1.0, unstable). When these pages and the SPEC disagree, the SPEC wins. [docs/architecture.md](https://github.com/tomiok/persiaDB/blob/main/docs/architecture.md) is the map of the code for contributors. Persia DB is an embedded, single-file, cloud-native search database, written in Rust. Think “SQLite for search”: open a file, index documents, and get ranked results that, by default, never come back empty. The same segment bytes can live in a local file or in object storage (S3, GCS, Azure Blob) and can be served by stateless query nodes on Kubernetes. A longer definition of the embedded library — in your process, one file, no search server — is [What is an embedded search database?](https://tomiok.github.io/persiaDB/guide/embedded-search/). Where the design sits next to SQLite FTS5, Tantivy, and Meilisearch is [Comparison](https://tomiok.github.io/persiaDB/guide/comparison/). That page is not a benchmark. Persia DB has no `persia::Db` search API yet (M10); the engine library code can search. How a typo is supposed to avoid an empty page is [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). Short answers on status, the file format, crash safety, SDKs, and the license are on the [FAQ](https://tomiok.github.io/persiaDB/guide/faq/). A local database is two files (SPEC §4, §9): * `.persia` — the index container (header and checksummed frames). * `.persia.blobs` — a content-addressed pack of client files (PDF, images, video), referenced from documents. The engine (index format, analysis, scoring, query execution) is written in this repository. [ADR-0001](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0001-own-engine-and-allowed-dependencies.md) records that decision and the closed list of allowed libraries. ## What works today [Section titled “What works today”](#what-works-today) The workspace exists, CI runs, and `persia-format` implements the byte-level primitives that later layers build on (ROADMAP M0 and M1, both done): * Bounds-checked little-endian readers and writers, canonical LEB128 varints, zigzag signed integers, and checked offset and length types (SPEC §4.1). * Integer codecs used inside future segment sections: delta coding, 128-value bit-packing, postings blocks, and frame-of-reference blocks for numeric columns (SPEC §5.2.1, [ADR-0004](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0004-integer-block-codecs.md)). * CRC32C, xxh3-64/128, and a helper for CRC-protected fixed-size structures (SPEC §4.1–§4.3). * Block compression codecs `None` and `Zstd`, with a decompression-bomb guard on every decode (SPEC §5.2, [ADR-0005](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0005-block-compression-codecs.md), ROADMAP 1.4). * Block framing of segment sections: per-block CRC and codec, and a block table that locates every block, so one block can be fetched and decoded on its own (SPEC §5.2, [ADR-0006](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0006-block-framing.md), ROADMAP 1.5). * The container header (magic, versions, feature flags, page size) and the two superblock slots, with the rules for picking the active slot (SPEC §4.2–§4.3, [ADR-0007](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0007-container-header-and-superblock.md), ROADMAP 1.6). * Frames: encode, decode, scan, and the streaming encoder, including a zstd payload flag and the classes of scan end (SPEC §4.4, [ADR-0009](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0009-frame-validation-and-scan-end.md), ROADMAP 1.7). The memory-map reader and `BlobId` / `BlobRef` are in the same crate. Those pieces are library code. persia-format implements the on-disk codecs (M1 done, 53/53). persia-storage implements the Storage trait, memory and local-file backends, and the single-file container (M2 done, 46/46). S3, GCS, and Azure have not been started (M12). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. persia-analysis implements tokenizers and the built-in analyzers de, en, es, fr, it, keyword, pt, standard, and whitespace (M4 done, 29/29). It does not search. persia-blob exports no API (M9 not started; its dependencies are done). The public persia crate exports no Db and no Collection. The crates are unpublished (publish = false, version 0.0.0). See [Getting started](https://tomiok.github.io/persiaDB/guide/getting-started/) and [Milestones](https://tomiok.github.io/persiaDB/guide/roadmap/). The first release a caller can use, `v0.1`, is ROADMAP M10. The format freezes at `v1.0` (M15). ## Why [Section titled “Why”](#why) Five properties the design does not trade away: 1. **Zero friction.** `Db::open("app.persia")` just works. No daemon and no config file for the embedded library. Server configuration, when it exists, is flags, then `PERSIA_*` environment variables, then an optional TOML file, with production-safe defaults (SPEC §14). Planned: M10, M11. 2. **Own engine.** Index format, analysis, BM25F scoring, and query execution live in this repository, behind a small allowed-dependency list (ADR-0001). 3. **Never zero results by default.** A query that matches nothing is relaxed step by step, and the response says exactly what changed (SPEC §7.6). The cascade is not started (M8). The query compiler and the BM25F scorer are library code. See [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/) and [Feature reference](https://tomiok.github.io/persiaDB/guide/features/). 4. **Cloud-native.** Segments are immutable. A local commit and an object-storage commit publish the same segment bytes. Query nodes are stateless. Planned: M12, M13. 5. **Correctness, then speed.** Checksums, crash-safe commits, and fuzzed parsers come before performance claims. SPEC §16.2 lists benchmark targets and labels them as targets, not promises. Scalar codec baselines already recorded live in [`docs/benchmarks.md`](https://github.com/tomiok/persiaDB/blob/main/docs/benchmarks.md). ## Goals and non-goals [Section titled “Goals and non-goals”](#goals-and-non-goals) Goals (SPEC §1.1): embedded search with an optional server; BM25F plus filters, facets, and zero-results avoidance; the same segment bytes locally or in object storage; horizontal scale by shard; content-addressed blobs; the embedded Rust API, embedded Go and TypeScript bindings, and Rust and Go network clients; a power loss at any point leaves committed data intact. Out of scope for v1 (SPEC §1.2): * Text extraction, OCR, and transcription. Clients supply text; blob bytes stay opaque. * Vector or semantic search. The format reserves a feature flag for a later layout (SPEC §18.6). * SQL, joins, multi-document transactions, and cross-collection queries. * More than one writer on the same shard. A lease enforces a single writer (SPEC §11.3). * Encryption at rest inside the file format. That stays with the disk or the storage service. The types that already exist are quoted on the [Code](https://tomiok.github.io/persiaDB/guide/code/) page. # What is an embedded search database? > What an embedded search database is: in-process full-text search, one file, and no server. When to choose it, and how Persia DB's Rust design approaches it. > **Status: pre-alpha.** Persia DB is pre-alpha. The public persia crate has no Db::open. persia-engine can put, delete, and commit a document. persia-engine runs a ranked search over a collection snapshot: BM25F, filters, sorts, facets, and pages. `persia::Db::open` and `search` elsewhere on this site are illustrative, from [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md). Counts below are read from [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) when the site is built: 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. That file is the live counter. When this page and the SPEC disagree, the SPEC wins. An **embedded search database** is a search library that runs in the same process as the application. You link it in. Queries are function calls in that process. There is no search daemon to install, and no separate server process that has to be up before a query can run. Some embedded libraries keep the index in one file, the way SQLite does. Others keep a directory of segment files. Persia DB’s design is the single-file kind. A local database is `.persia`, plus an optional `.persia.blobs` pack ([SPEC §4](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#4-local-file-container), [§9](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#9-blobs)). The planned call is `Db::open("app.persia")`, with no config file for the library ([SPEC §14](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#14-configuration)). That call is not in the crate. **Embedded full-text search in Rust** is that in-process shape, as a Rust crate. [Tantivy](https://github.com/quickwit-oss/tantivy) is a released library of that kind, and its index is a directory. Persia DB is a pre-alpha project aimed at one file, an engine written in this repository, and the same segment bytes in object storage. The design line is “SQLite for search”: open a file, index documents, and get ranked results that, by default, never come back empty ([SPEC §1](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#1-goals-and-non-goals)). The repository has no `persia::Db` search API yet (M10); the engine library code can search. How that empty-page rule is designed, and which query pieces exist, is [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). A search server is a different way to run search. You start it as its own process, send documents and queries over the network, and operate that service. [Meilisearch](https://www.meilisearch.com/docs) and [Elasticsearch](https://www.elastic.co/elasticsearch) are servers. [SQLite FTS5](https://sqlite.org/fts5.html) and [Tantivy](https://github.com/quickwit-oss/tantivy) are libraries. Where Persia DB sits next to SQLite FTS5, Tantivy, and Meilisearch is the [comparison](https://tomiok.github.io/persiaDB/guide/comparison/). That page is not a benchmark. ## When an embedded library fits [Section titled “When an embedded library fits”](#when-an-embedded-library-fits) The choice is about where the index lives and who operates it. An embedded search database fits when search belongs in the same program as the data: * The program is one process (a CLI, a desktop tool, a single service, a test) and can open a file or a directory. * The same binary that writes documents runs the query. * For the single-file kind, the database you copy is that file. In the Persia DB design it is `.persia`, plus `.persia.blobs` when blobs are used (SPEC §4, §9). * You do not want a search cluster for that data. A search server fits when many applications share one index, when callers should use HTTP without linking a library, or when a team already runs that service. The Persia DB design is the library, with a server only as a later option. The planned gRPC and HTTP server embeds the same engine as the library ([SPEC §3](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#3-architecture-overview), [§12](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#12-server-and-api)). That server is milestone M11. `persia-server` is a placeholder binary that exits with “not implemented (M11)”. See [Server, cloud, and Kubernetes](https://tomiok.github.io/persiaDB/guide/server/). For a search feature in production today, use an engine that already answers queries. The [comparison](https://tomiok.github.io/persiaDB/guide/comparison/) says which established tool fits which job. Persia DB has no `persia::Db` search API yet (M10); the engine library code can search. ## How the Persia DB design approaches it [Section titled “How the Persia DB design approaches it”](#how-the-persia-db-design-approaches-it) The design contract is [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md) (draft v0.1). The table is that design. The right-hand column is what the repository contains. | Design choice | Specified in | In the repository | | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | In-process library. No daemon and no config file for the embedded API. Options are builders, with defaults. | [SPEC §14](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#14-configuration) | Planned (M10). The `persia` crate exports no `Db` and no `Collection`. | | One local file, `.persia`, plus an optional `.persia.blobs` pack for client bytes such as PDF or images. | SPEC §4, §9 | The container format and the single-file container (open, append, commit, recover, compact) are implemented (M1, M2). Done (M5, 48/48). Segments are searched through CollectionViews::search. Container commit is done (M2, 46/46). The document write path is in progress (M6, 49/58): WAL, flush, commit, and snapshots, deletes and upserts, and segment merging. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle. The blob store has not been started (M9). `BlobId` and `BlobRef` exist as types in `persia-format`. | | The engine is written in this repository: analysis, BM25F, filters, facets, and query execution. | SPEC §6, §7. [ADR-0001](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0001-own-engine-and-allowed-dependencies.md) forbids depending on Tantivy, Lucene, or SQLite. | persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. | | A query with too few hits is relaxed step by step (synonyms, dropped terms, prefix, a fuzzy step, then a fallback). The response names the step. On by default. | [SPEC §7.6](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#76-zero-results-avoidance-default-on) | Not started (M8). Ranking and relaxation settings exist as schema config structs. The cascade does not run. See [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). | | Segments are immutable. The same segment bytes live in a local file or in S3, GCS, or Azure Blob. Query nodes are stateless. | SPEC §3, §5, §10, §11 | A segment writer and reader exist (M5). Memory and local-file storage exist (M2). Object-storage backends and Kubernetes are planned (M12, M13). | | A power loss leaves the previous commit or the new one. | [SPEC §4.5](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#45-commit-protocol-local), §8 | The container commit is done (M2, 46/46) and is crash-tested. The engine write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Database::open reloads the committed manifest, checks it against the superblock, and replays the WAL. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle. | | Optional gRPC and HTTP server. Embedded mode has no authentication. | SPEC §12, §14 | Planned (M11). | | Embedded Go and TypeScript bindings open the same file in-process. Rust and Go network clients talk to the server. | SPEC §13, [ADR-0018](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0018-embedded-sdk-bindings.md) | Design only. No Go module and no npm package. Network clients are milestone M14. | The first release a caller is supposed to use is v0.1 (M10). That milestone is blocked on M6 (in progress), M7 (in progress), M8 (not started), and M9 (not started). The on-disk format stays unstable until `v1.0` (M15, [SPEC §15](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#15-versioning-and-compatibility)). The crates are unpublished (`publish = false`, version 0.0.0). There is no crates.io release. [SPEC §16.2](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#16-quality-requirements) lists latency and throughput figures and labels them as targets, not promises. This project does not publish search benchmarks. There is no `Db::search`. The query compiler, the BM25F scorer, and matching with score top-k over one view are library code (ROADMAP 7.1–7.4.4). So are filter evaluation, sorts, and facets, through `CollectionViews::search` over a collection snapshot (ROADMAP 7.4.5, 7.5). A search call on the database handle is not. Out of scope for v1 ([SPEC §1.2](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#12-non-goals-v1)): text extraction, OCR, and transcription (clients supply text; blob bytes stay opaque); vector or semantic search; SQL, joins, multi-document transactions, and cross-collection queries; more than one writer on a shard; encryption at rest inside the file. ## What the code can do today [Section titled “What the code can do today”](#what-the-code-can-do-today) [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) is the live counter: 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. Milestone states are on the [roadmap](https://tomiok.github.io/persiaDB/roadmap/) page, generated from that file. persia-format implements the on-disk codecs (M1 done, 53/53). persia-storage implements the Storage trait, memory and local-file backends, and the single-file container (M2 done, 46/46). S3, GCS, and Azure have not been started (M12). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. persia-analysis implements tokenizers and the built-in analyzers de, en, es, fr, it, keyword, pt, standard, and whitespace (M4 done, 29/29). It does not search. persia-blob exports no API (M9 not started; its dependencies are done). The public persia crate exports no Db and no Collection. The crates are unpublished (publish = false, version 0.0.0). Building one segment is not `persia::Db::open`. A git dependency can use: * `persia-format`, which encodes and decodes on-disk bytes: integer codecs, checksums, `None` and `Zstd` block compression, block framing, the container header, the superblock, and frames. * `persia-storage`, which implements the `Storage` trait, a memory backend, a local-file backend, and the single-file container. * persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. The `persia` command lists and removes temporary and quarantine files next to a container or a local store (`persia container files`, `persia container sweep`, `persia store files`, `persia store sweep`). It can also inspect and dump one segment file (`persia inspect segment`, `persia dump segment`, with `--verify` on inspect). It does not import documents or search. There is no `persia::Db::open`. The intended API, marked as illustrative, is the [embedded API](https://tomiok.github.io/persiaDB/guide/embedded-api/) page. The file layout is [File format and storage](https://tomiok.github.io/persiaDB/guide/file-format/). ## Read next [Section titled “Read next”](#read-next) * [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/) — query relaxation, BM25F, and filters: design versus what the engine implements. * [Comparison](https://tomiok.github.io/persiaDB/guide/comparison/) — Persia DB next to SQLite FTS5, Tantivy, and Meilisearch. Not a benchmark. * [FAQ](https://tomiok.github.io/persiaDB/guide/faq/) — short answers on status, the file format, crash safety, SDKs, and the license. * [How it works](https://tomiok.github.io/persiaDB/architecture/) — crate layers, the write path, and the query path. * [Roadmap](https://tomiok.github.io/persiaDB/roadmap/) — milestone status. The live counter is PROGRESS.md. The longer dependency notes are [Milestones](https://tomiok.github.io/persiaDB/guide/roadmap/). * [Overview](https://tomiok.github.io/persiaDB/guide/overview/) — the project narrative, including goals and non-goals. * [Crate architecture](https://tomiok.github.io/persiaDB/guide/architecture/) — how the Rust crates are layered. # Typo-tolerant search and zero results > How Persia DB designs typo-tolerant embedded search in Rust: query relaxation so a fuzzy query avoids zero results, and where BM25F and filters fit today. > **Status: pre-alpha.** Persia DB is pre-alpha. The public persia crate has no Db::open. persia-engine can put, delete, and commit a document. persia-engine runs a ranked search over a collection snapshot: BM25F, filters, sorts, facets, and pages. Counts below are read from [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) when the site is built: 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. When this page and [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md) disagree, the SPEC wins. Nothing here is a benchmark. Typo-tolerant search, in this design, is not a separate fuzzy index. A query that would otherwise come back empty is loosened one step at a time until it has enough hits. That cascade is [query relaxation](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#76-zero-results-avoidance-default-on) (SPEC §7.6). It is on by default. Fuzzy edit distance is one step of it, after an exact match, synonyms, and dropping terms. The product goal is embedded full-text search in Rust that, by default, never returns zero results ([SPEC §1](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#1-goals-and-non-goals)). What that library shape is, and what you can call today, is [What is an embedded search database?](https://tomiok.github.io/persiaDB/guide/embedded-search/). This page is the search behavior: how a typo is supposed to be handled, how [BM25F](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#72-scoring-bm25f) and filters fit, and which of those pieces are in the repository. ## What typo-tolerant search means here [Section titled “What typo-tolerant search means here”](#what-typo-tolerant-search-means-here) A user types `runing shoes` and expects shoes, not an empty page. The specified behavior is: 1. Run the query as written. Every analyzed term is required (default AND). 2. If the hit count is below `min_results` (default 1), try the next relaxation level. 3. Stop at the first level that has enough hits, unless `fill` is set, in which case looser levels may complete the page. 4. Rank hits from a stricter level above hits from a looser level. 5. Say what changed: the level, the steps applied, the original text, the effective text, and any corrected terms. Filters stay as the caller wrote them. A brand or a price range is not dropped to manufacture a hit, unless the query sets `relax.filters` to `"last_resort"`. A query can turn the cascade off with `relax: false`. [Meilisearch](https://www.meilisearch.com/docs) is a released server whose typo tolerance is on by default. Persia DB’s planned default is this cascade, in a library, with the response naming the step. The mechanisms differ. That comparison, which is not a benchmark, is [Persia DB vs SQLite FTS5, Tantivy, and Meilisearch](https://tomiok.github.io/persiaDB/guide/comparison/). The cascade has not been started. Milestone M8 (zero-results avoidance) is blocked on the query milestone and has no leaves done. ROADMAP 8.1 through 8.4 — the orchestrator, the levels, the response metadata, and the zero-result regression gate — are unchecked. `RelaxationConfig` in `persia-engine` stores the collection defaults from SPEC §6.2.3 (`enabled`, `min_results`, `levels`, `fallback`, `fill`, `filters`, `budget_ms`). Schema JSON parses them. Nothing walks the levels at search time. ## The relaxation cascade [Section titled “The relaxation cascade”](#the-relaxation-cascade) Each level is specified in SPEC §7.6. The names are the ones the response and the schema use. | Level | Name | What it changes | In the tree | | ----- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | L0 | `exact` | Every analyzed term is required. | The user-text compiler builds this shape (ROADMAP 7.1). Nothing matches it against postings. | | L1 | `synonyms` | A term expands through the collection’s synonym groups. Within a group the terms are alternatives. Expansions per matched term are capped (default 50). | `SynonymMap` in `persia-analysis` looks up analyzed tokens (ROADMAP 4.2.5). The search level that would call it (ROADMAP 8.2.1) is not written. | | L2 | `drop_terms` | The share of terms that must match steps from 100% to 75%, to 50%, then to one term. Low-IDF terms go first. | Not started (ROADMAP 8.2.2). | | L3 | `prefix` | The last term, and other terms of length at least 3, expand through an FST prefix scan. Expansions are capped (default 50). | The AST has a `Prefix` node. No scan runs (ROADMAP 8.2.3). | | L4 | `fuzzy` | Edit distance 1 for a term of length 4–7, distance 2 for length 8 or more, via a Levenshtein automaton intersected with the term dictionary. Corrected terms are reported. | The AST has a `Fuzzy` node with distance 1 or 2. No automaton runs (ROADMAP 8.2.4). | | L5 | `fallback` | Top documents by static rank, or by recency when the collection has no static rank. `none` allows an empty page. | Not started (ROADMAP 8.2.5). | Other rules from the same section, all still design: * A time budget, `budget_ms`, defaults to 50 ms. Past it, the best results so far come back with `truncated: true`. That flag is distinct from `query_truncated`, which means the compiler cut the user’s text to fit the query limits (SPEC §7.1.1). * Holding the queries a level builds to those same limits is pending (ROADMAP 8.1.8, SPEC §7.1.3). A level is specified to drop the least important expansions first and to name that cut in `applied`. * The response object below is the shape in SPEC §7.6. No function returns it. ```json { "level": "fuzzy", "applied": ["drop_terms:75%", "fuzzy"], "original": "zapatilas rojs", "effective": "zapatillas rojas", "corrected": { "zapatilas": "zapatillas", "rojs": "rojas" } } ``` The same table, and the schema JSON that carries `relaxation`, is the [feature reference](https://tomiok.github.io/persiaDB/guide/features/#zero-results-relaxation). The diagram on [Features](https://tomiok.github.io/persiaDB/features/#zero-results-relaxation) is this cascade. ## Compiling a query [Section titled “Compiling a query”](#compiling-a-query) Before any relaxation, user text becomes a query (SPEC §7.1, §7.1.1). That compiler is in the tree (ROADMAP 7.1). It does not search. `QueryCompiler::compile` never fails on user text. Syntax that does not parse is read as plain text. Clauses are split on whitespace and all required (default AND). A clause is an optional `-`, an optional `field:`, and a word or a `"phrase"`. `-` excludes the documents the clause would match. `field:` limits it to that `text` or `keyword` field. A `"` without a closing quote is an ordinary character. Text past `max_text_len` (default 8 KiB) is left out on a clause boundary, so a cut word does not become a required term that matches nothing. With no clause left, the query is `match_all`. Words are analyzed with each field’s query-time analyzer. Fields that agree on token spans become one `combined` clause per span: one query word, several fields, scored as one BM25F term. A phrase on a field with positions stays a phrase. On a field without positions it becomes a boolean that requires each token. Filters are a separate JSON object (SPEC §7.1.2): `term`, `terms`, `range`, `exists`, `and`, `or`, and `not`. They are written by programs, so a bad filter is an error. User text is not. `Filter::from_json` parses and validates them against the schema. Nothing yet evaluates a filter against a segment (ROADMAP 7.4.5). Limits (SPEC §7.1.3), implemented as `QueryLimits`, reject a hand-built query or a filter that exceeds them. Defaults: 1024 leaf terms, 1024 clauses, 1024 expansions summed across prefix and fuzzy clauses, depth 32, and 8 KiB of `q`. The prefix and fuzzy clauses the engine itself would build use a per-clause cap of 50 (`DEFAULT_MAX_EXPANSIONS`). This is library code in `persia-engine`. It compiles text. It does not open a database and it does not return hits: ```rust use persia_analysis::AnalyzerRegistry; use persia_engine::query::{DefaultFields, Filter, QueryCompiler, QueryLimits}; use persia_engine::schema::{FieldKind, KeywordOptions, SchemaBuilder, TextOptions}; let schema = SchemaBuilder::new("products") .field("title", FieldKind::Text(TextOptions { positions: true, ..TextOptions::default() })) .field("brand", FieldKind::Keyword(KeywordOptions::default())) .field("price", FieldKind::F64 { stored: true }) .build()?; let compiler = QueryCompiler::new(&schema, &AnalyzerRegistry::builtin())?; let limits = QueryLimits::default(); let compiled = compiler.compile( r#"red "running shoes" -kids"#, &DefaultFields::all(&schema), &limits, ); compiled.query.validate(&schema, &limits)?; let filter = Filter::from_json( r#"{"range": ["price", {"lte": 100}]}"#, &schema, &limits, )?; ``` The AST can also represent `prefix` and `fuzzy` clauses. `Query::validate` accepts a fuzzy distance of 1 or 2. The compiler does not turn a typo into one of those clauses. A typo stays the term the analyzer produced, and under default AND it is required. Fuzzy matching is L4 of the cascade, which is not written. An explicit `prefix` or `fuzzy` clause plans with `SearchPlan::with_terms`, which expands it over the collection terms (ROADMAP 7.4.12). Matching, sorts, pages, and facets over a collection snapshot are library code (`CollectionViews::search`). There is no search call on the database handle. ## BM25F scoring and filters [Section titled “BM25F scoring and filters”](#bm25f-scoring-and-filters) Ranking is BM25F (SPEC §7.2). For a term `t` and a document `d`, over fields `f` with weights `w_f`: ```text tf~(t,d) = Σ_f w_f · tf(t,d,f) / (1 − b_f + b_f · len_f(d) / avglen_f) score(t,d) = idf(t) · tf~ · (k1 + 1) / (k1 + tf~) idf(t) = ln(1 + (N − df + 0.5) / (df + 0.5)) ``` `N`, `df`, and `avglen` are collection-level. `TermScorer` computes one query word from those statistics plus a document’s term frequency and norm byte in each field. A `combined` clause is one word over several `text` fields; `df` is the max of the per-field document frequencies. A `term` on a `keyword` field uses weight 1, `b` 0, and `k1` 1.2. The total adds a static rank when the schema has one: `weight · log1p(value)` or `weight · linear(value)` (SPEC §6.2.3). `must` and `should` clauses score. `filter` and `must_not` clauses do not, so a filter never inflates BM25F. `StatisticsProvider` is the trait those numbers come from. `SegmentStatistics` sums the segments of one snapshot, deleted documents included until a merge drops them. `SearchStatistics` adds the memtable’s live documents. A snapshot computes the segment part once. The same function, given a block’s maximum term frequency and minimum norm, is an upper bound used by MaxScore and block-max WAND (SPEC §7.3, ROADMAP 7.3.3 and 7.4.4) to skip documents that cannot reach the top k. `persia-engine/tests/reference` evaluates the formula on the quantized field lengths. The scorer must match it within `1e-9 · idf·(k1+1)` (SPEC §7.2, ROADMAP 7.3.4). A differential suite over random corpora and full queries is ROADMAP 7.8.1 and is not written. [SPEC §16.2](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#16-quality-requirements) lists latency figures and labels them as targets, not promises. This project does not publish search benchmarks. What scoring does not do: walk postings, apply tombstones, collect a top-k, facet, or sort. A score for one document you already hold is not a search result. ## What runs today [Section titled “What runs today”](#what-runs-today) | Piece | Status | | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | User-text compiler, query AST, filter DSL, query limits | Library code (ROADMAP 7.1). | | Collection statistics | Library code (ROADMAP 7.2). | | BM25F scorer, static rank, score upper bounds | Library code (ROADMAP 7.3), checked against the reference scorer. | | `RelaxationConfig` and `SynonymMap` | Stored and looked up. Not called from a search. | | Term, phrase, boolean, prefix, and fuzzy matching | Library code (M7 in progress, 46/51): matching, sorts, pages, and facets. | | Filter evaluation, facets, sort, explain | Library code (M7 in progress, 46/51): matching, sorts, pages, and facets. Explain is library code (ROADMAP 7.7). | | Relaxation cascade | Not started (M8). On by default in the design. | | `persia::Db::open` and `search` | Not in the crate (M10). | `persia_engine::db::Database` has `open`, `put`, and `commit`; searching goes through `CollectionViews::search` on a snapshot, not through `Database`. The public `persia` crate exports no `Db`. persia_engine::db::Database has open, create_collection, put, put_batch, delete, and commit. CollectionSnapshot::get fetches one document by id. CollectionViews::search runs a ranked search over a collection snapshot. There is no persia::Db, no search on the database handle, no put_struct, no put_blob, no get_blob, and no OpenOptions. Those SPEC names are not the public embedded API. The call below is the intended API from SPEC §7.1 and §7.6. It is illustrative: these methods are not in the `persia` crate. ```rust // ILLUSTRATIVE — these methods are not in the persia crate. let db = persia::Db::open("app.persia")?; let products = db.create_collection(schema)?; products.put(doc)?; products.commit()?; // A typo is specified to return hits. The response says which level ran. let results = products.search("runing shoes")?; let relaxation = &results.relaxation; ``` [Getting started](https://tomiok.github.io/persiaDB/guide/getting-started/) separates that sketch from the crates you can depend on. [Code](https://tomiok.github.io/persiaDB/guide/code/) links the modules that exist. The planned method list is the [embedded API](https://tomiok.github.io/persiaDB/guide/embedded-api/). ## FAQ [Section titled “FAQ”](#faq) ### Does Persia DB do typo-tolerant search today? [Section titled “Does Persia DB do typo-tolerant search today?”](#does-persia-db-do-typo-tolerant-search-today) No. The relaxation cascade has not been started (M8). A typo in user text stays a required term. The query AST can represent a fuzzy clause, and `Query::validate` accepts an edit distance of 1 or 2, and an explicit fuzzy or prefix clause expands over the collection terms as library code (ROADMAP 7.4.12). The compiler does not turn a typo into one. A ranked search over a collection snapshot is library code (CollectionViews::search). There is no search call on the database handle. ### What is query relaxation? [Section titled “What is query relaxation?”](#what-is-query-relaxation) The cascade in SPEC §7.6. If a query has fewer hits than `min_results` (default 1), the engine is specified to walk exact match, synonyms, dropped terms, prefix, fuzzy edit distance, then a fallback, and to stop at the first level with enough hits. The response names the level. That cascade is not implemented. `RelaxationConfig` stores the collection defaults. ### Are filters relaxed when a search has zero results? [Section titled “Are filters relaxed when a search has zero results?”](#are-filters-relaxed-when-a-search-has-zero-results) Not by default. SPEC §7.6 keeps filters strict unless `relax.filters` is `"last_resort"`. The filter DSL parses and validates (`term`, `terms`, `range`, `exists`, `and`, `or`, `not`). Evaluating a filter against documents is ROADMAP 7.4.5 and has not been started. Filter clauses do not add to the BM25F score. ### Is BM25F scoring implemented? [Section titled “Is BM25F scoring implemented?”](#is-bm25f-scoring-implemented) The formula is. `TermScorer` scores one query word from a document frequency, a term frequency, and a norm byte, using collection statistics. Static rank and score upper bounds are in the same module. Tests check it against a reference scorer. It does not walk postings or return a ranked hit list. This project does not publish search benchmarks. ### Can I call Db::open and search for a fuzzy query? [Section titled “Can I call Db::open and search for a fuzzy query?”](#can-i-call-dbopen-and-search-for-a-fuzzy-query) No. `persia::Db` is not in the crate, and `Database` has no `search` method. `Database::open`, `put`, and `commit` exist. Code samples that call `Db::search` on this site are illustrative. ## Read next [Section titled “Read next”](#read-next) * [What is an embedded search database?](https://tomiok.github.io/persiaDB/guide/embedded-search/) — in-process full-text search, one file, and no server. * [Comparison](https://tomiok.github.io/persiaDB/guide/comparison/) — next to SQLite FTS5, Tantivy, and Meilisearch. Not a benchmark. * [Feature reference](https://tomiok.github.io/persiaDB/guide/features/) — the cascade table, the schema sketch, and the status of each capability. * [Features](https://tomiok.github.io/persiaDB/features/) — the short status table, including the relaxation diagram. * [Getting started](https://tomiok.github.io/persiaDB/guide/getting-started/) — crates you can call, and the API that is still illustrative. * [FAQ](https://tomiok.github.io/persiaDB/guide/faq/) — production readiness, the file format, crash safety, and the license. * [How it works](https://tomiok.github.io/persiaDB/architecture/) — the query path from analyzed text to the cascade. * [Roadmap](https://tomiok.github.io/persiaDB/roadmap/) — milestone status. The live counter is PROGRESS.md. # Page not found > This page is not on the Persia DB site. That address is not a page on this site. * [Home](https://tomiok.github.io/persiaDB/) * [Guide overview](https://tomiok.github.io/persiaDB/guide/overview/) * [GitHub repository](https://github.com/tomiok/persiaDB) # About > Who Persia DB is for: an open-source embedded search database written in Rust, how to contribute, and the Apache-2.0 license. Persia DB is an open-source search database. The aim is a library you can embed the way you embed SQLite, with a file format that also works from object storage and from stateless query nodes. It is pre-alpha. The first release a caller can use is `v0.1` (roadmap M10). The format freezes at `v1.0` (M15). Search in production today means an existing engine. [Comparison](https://tomiok.github.io/persiaDB/guide/comparison/) places this design next to SQLite FTS5, Tantivy, and Meilisearch. Copyright 2026 The Persia DB Authors. See [NOTICE](https://github.com/tomiok/persiaDB/blob/main/NOTICE). ## Community [Section titled “Community”](#community) Contributions are welcome, and the bar is correctness first. Pick one leaf from [ROADMAP.md](https://github.com/tomiok/persiaDB/blob/main/ROADMAP.md). [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) lists what is ready next. One leaf is one pull request. If the SPEC is unclear, open an issue before writing the code. By participating you agree to the [Code of Conduct](https://github.com/tomiok/persiaDB/blob/main/CODE_OF_CONDUCT.md). Report security issues privately, as described in [SECURITY.md](https://github.com/tomiok/persiaDB/blob/main/SECURITY.md). Only the latest commit on `main` receives fixes. A panic or unbounded memory use on a crafted database file is a security issue. The repository also documents how AI-assisted changes follow the same rules, in [CONTRIBUTING.md](https://github.com/tomiok/persiaDB/blob/main/CONTRIBUTING.md). [Contributing guide](https://tomiok.github.io/persiaDB/guide/contributing/)[Repository](https://github.com/tomiok/persiaDB)[License](https://tomiok.github.io/persiaDB/guide/license/) ## License [Section titled “License”](#license) Persia DB is licensed under the [Apache License, Version 2.0](https://github.com/tomiok/persiaDB/blob/main/LICENSE). Contributions are licensed under the same terms, as stated in section 5 of the license. # How it works > How a Persia DB write and query move through the engine: crate layers, the write path, the query path, and segment storage. Persia is a library you open on a file, plus a server that embeds the same engine. That library shape is an [embedded search database](https://tomiok.github.io/persiaDB/guide/embedded-search/): full-text search in the application process, stored in a file. The engine is synchronous. It talks to disks and object storage only through the `Storage` trait. Async belongs in the cloud backends and in the server. Memory and local-file backends exist. S3, GCS, and Azure do not. ``` ╔═ layers ═════════════════════════════════════════════════════════╗ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-server, Rust SDK, Go SDK ┃ ║░ ║ ┃ planned ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia · Db and Collection ┃ ║░ ║ ┃ stub · no methods yet ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ engine ┃ ┃ blob ┃ ┃ analysis ┃ ║░ ║ ┃ WAL and query ┃ ┃ chunks, dedup ┃ ┃ analyzers ┃ ║░ ║ ┃ in progress ┃ ┃ not started ┃ ┃ done ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-storage ┃ ║░ ║ ┃ memory and local file ┃ ║░ ║ ┃ s3, gcs, azure planned ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-format ┃ ║░ ║ ┃ encodings, checksums, codecs ┃ ║░ ║ ┃ done ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ╚══════════════════════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ``` Layers from [SPEC §3](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#3-architecture-overview). Status from the 2026-11-05 PROGRESS.md snapshot. Format, local storage, schema, and text analysis are done (M1–M4). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. The blob store is not started (M9; its dependencies are done). The embedded API has not been started (M10), and the persia crate exports no Db or Collection. The server is not started (M11). Network SDKs are not started (M14). S3, GCS, and Azure are not started (M12). Arrows point down: each layer is used by the one above it. ## A write [Section titled “A write”](#a-write) ``` ╔═ write ══════════════════════════════════════════════════════════╗ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ 1. put or delete ┃ ║░ ║ ┃ WAL, then memtable ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ 2. flush ┃ ║░ ║ ┃ memtable becomes a segment ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ 3. commit ┃ ║░ ║ ┃ data, then manifest, then ┃ ║░ ║ ┃ the inactive superblock ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ 4. merge, later ┃ ║░ ║ ┃ new segment, new manifest ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ╚══════════════════════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ``` Local write path, partly in the tree ([SPEC §4.5](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#45-commit-protocol-local), [§8.2](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#82-flow)). A crash before the superblock write keeps the previous commit. Deletions live in tombstone sidecars. The container commit is done (M2, 46/46) and is crash-tested. The engine write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Database::open reloads the committed manifest, checks it against the superblock, and replays the WAL. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle. ## A query [Section titled “A query”](#a-query) ``` ╔═ query ══════════════════════════════════════════════════════════╗ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ 1. analyze the text ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ 2. each segment ┃ ║░ ║ ┃ and the memtable ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ 3. merge the top hits ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ 4. too few hits? ┃ ║░ ║ ┃ no: return them ┃ ║░ ║ ┃ yes: relax ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ 5. cascade, L0 to L5 ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ╚══════════════════════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ``` Designed query path ([SPEC §7](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#7-search)). The text compiler in step 1 and the BM25F scorer are in the tree. Searching segments, merging hits, and the cascade are not. Step 4 is the branch into zero-results relaxation. Filters stay strict unless the caller opts in. The response names the level and what changed. The longer account is [typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). Text analysis is done (M4, 29/29). The query milestone is in progress (M7, 46/51). Zero-results relaxation is not started (M8). The compiler and the BM25F scorer are in the tree. Searching segments and the relaxation cascade are not. [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/) is the longer account. ## Storage and segments [Section titled “Storage and segments”](#storage-and-segments) A segment is one byte string: sections, then a footer. Locally it is a frame payload. Remotely it is one object. The bytes match. Codecs on a block are `None` and `Zstd`. The local container, the block codecs, and the memory and file backends exist. persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. S3, GCS, and Azure backends do not (M12). ``` ╔═ storage ════════════════════════════════════════════════════════╗ ║ local file object storage ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ header, 12 KiB ┃ same bytes ┃ manifests/seq ┃ ║░ ║ ┃ two superblocks ┃ ┃ segments/... ┃ ║░ ║ ┃ append-only frames ┃ ┃ blobs/... ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ sections, then sections ┃ ║░ ║ ┃ footer: stats, table, CRC ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ 16 KiB blocks · codec None or Zstd ║░ ╚══════════════════════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ``` The same segment bytes in a local frame and in object storage ([SPEC §5](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#5-segment-format), [§10](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#10-storage-abstraction-and-cloud-backends)). Deleted ids are not a section. Block framing is done (M1, 53/53). Memory and local-file backends are done (M2, 46/46). S3, GCS, and Azure are not started (M12). The contributor map, with the crate dependency graph, is [docs/architecture.md](https://github.com/tomiok/persiaDB/blob/main/docs/architecture.md). Quotes from the code that exists today are on the [code](https://tomiok.github.io/persiaDB/guide/code/) page. [Crate architecture](https://tomiok.github.io/persiaDB/guide/architecture/)[Read the code](https://tomiok.github.io/persiaDB/guide/code/)[File format](https://tomiok.github.io/persiaDB/guide/file-format/) # Features > What Persia DB is designed to do, and which of those features exist today. The product is a search database: ranked full-text search, filters, facets, and a default that avoids an empty page. Ranked full-text search, filters, sorts, and facets run as library code in persia-engine, over a collection snapshot. There is no Db to open yet (M10). Relaxation is not started (M8), so a search that matches nothing returns no hits. The write path is in progress (M6, 49/58). The table says which milestone builds each row. The long form, including the schema sketch and the relaxation JSON, is the [feature reference](https://tomiok.github.io/persiaDB/guide/features/). How a typo is supposed to avoid an empty page is [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). | Capability | Status | | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Open a file and search | Not started (M10). Below it, persia_engine::db::Database opens a file and CollectionViews::search searches it. | | Query text, filters, and BM25F | Library code (M7 in progress, 46/51): the compiler, the filter DSL, and the scorer return ranked hits through CollectionViews::search. | | Matching, facets, and sort | Library code (M7 in progress, 46/51): matching, sorts, pages, and facets. | | Zero-results relaxation | Not started (M8). On by default in the design. | | Schema and documents | Done as library code (M3). Not reachable through `Db`. | | Analyzers | Done as library code (M4): `standard`, `keyword`, `whitespace`, `en`, `es`, `pt`, `fr`, `de`, `it`. The query compiler runs them. | | Segment writer and reader | Done (M5, 48/48). Segments are searched through CollectionViews::search. | | Crash-safe commit and recovery | Container commit is done (M2, 46/46). The document write path is in progress (M6, 49/58): WAL, flush, commit, and snapshots, deletes and upserts, and segment merging. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle. | | Blob storage | Not started (M9; its dependencies are done). | | gRPC and HTTP server | Not started (M11). | | S3, GCS, Azure, replication | Not started (M12). | | Kubernetes and sharding | Not started (M13). | | Embedded Go and TypeScript bindings | Planned (after M10). Design only. | | Rust and Go network SDKs | Not started (M14). | | Integer codecs and `None` / `Zstd` block codecs | Done (M1, 53/53). Library code in `persia-format`. | ## Zero-results relaxation [Section titled “Zero-results relaxation”](#zero-results-relaxation) When a query matches nothing, the design walks a cascade and stops at the first level that returns enough hits. Stricter levels rank above looser ones. The response says what changed. Filters stay strict unless the caller opts in. ``` ╔═ relaxation ═════════════════════════════════════════════════════╗ ║ ┏━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ L0 exact ┃ every term required ║░ ║ ┗━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ L1 synonyms ┃ expand groups ║░ ║ ┗━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ L2 drop terms ┃ 100, 75, 50, 1 ║░ ║ ┗━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ L3 prefix ┃ capped expansions ║░ ║ ┗━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ L4 fuzzy ┃ edit distance 1 or 2 ║░ ║ ┗━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ L5 fallback ┃ static rank, or none ║░ ║ ┗━━━━━━━━━━━━━━━━┛ ║░ ╚══════════════════════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ``` The cascade in [SPEC §7.6](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#76-zero-results-avoidance-default-on). Stop at the first level with enough hits. A stricter level ranks above a looser one. Filters stay as written unless `relax.filters` is `last_resort`. The full table is in the [feature reference](https://tomiok.github.io/persiaDB/guide/features/#zero-results-relaxation). What is implemented, and what is still the design, is [typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). Out of scope for v1: text extraction and OCR, vector search, SQL and joins, more than one writer on a shard, and encryption inside the file format. Those are [SPEC §1.2](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#12-non-goals-v1). [Feature reference](https://tomiok.github.io/persiaDB/guide/features/)[How it works](https://tomiok.github.io/persiaDB/architecture/) # Crate architecture > How Persia DB crates are layered in Rust, from persia-format up to the server, and the rules for immutable segments and synchronous I/O. The picture below is the intended layering in [SPEC §3](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#3-architecture-overview). `persia-format`, `persia-storage`, `persia-engine`, and `persia-analysis` contain library code. persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. `persia` exports no `Db`. `persia-server` is a placeholder binary that exits with “not implemented (M11)”. `persia-blob` and `persia-proto` are placeholders, module docs only. [`docs/architecture.md`](https://github.com/tomiok/persiaDB/blob/main/docs/architecture.md) is the contributor map (crate graph, write path, query path) and stays the place for that detail. ``` ╔═ layers ═════════════════════════════════════════════════════════╗ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-server, Rust SDK, Go SDK ┃ ║░ ║ ┃ planned ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia · Db and Collection ┃ ║░ ║ ┃ stub · no methods yet ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ engine ┃ ┃ blob ┃ ┃ analysis ┃ ║░ ║ ┃ WAL and query ┃ ┃ chunks, dedup ┃ ┃ analyzers ┃ ║░ ║ ┃ in progress ┃ ┃ not started ┃ ┃ done ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-storage ┃ ║░ ║ ┃ memory and local file ┃ ║░ ║ ┃ s3, gcs, azure planned ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-format ┃ ║░ ║ ┃ encodings, checksums, codecs ┃ ║░ ║ ┃ done ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ╚══════════════════════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ``` Layers from [SPEC §3](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#3-architecture-overview). Status from the 2026-11-05 PROGRESS.md snapshot. Format, local storage, schema, and text analysis are done (M1–M4). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. The blob store is not started (M9; its dependencies are done). The embedded API has not been started (M10), and the persia crate exports no Db or Collection. The server is not started (M11). Network SDKs are not started (M14). S3, GCS, and Azure are not started (M12). Arrows point down: each layer is used by the one above it. Three rules shape the rest of the design: 1. **Immutable segments and append-only metadata.** Crash recovery, caching, and replication all read a snapshot instead of locking a mutable index. 2. **The same logical objects everywhere.** A segment, manifest, WAL chunk, or blob is a named object. Locally they are frames in one file; remotely they are one object each. The segment bytes do not change between the two. 3. **The engine is synchronous.** It talks to I/O only through the `Storage` trait. Async belongs in the cloud backends and in the server. | Crate | Role | Status | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `persia-format` | Encodings, checksums, frames, block codecs. No I/O policy. | Implemented (M1) | | `persia-storage` | `Storage` trait, memory, local file, later S3 / GCS / Azure. | Memory, local file, and the container are implemented (M2). Cloud is M12. | | `persia-analysis` | Tokenizers, normalizers, stemmers, analyzers. | Done (M4). The query compiler runs them. This crate does not search. | | `persia-engine` | Schema, segments, WAL, commit, merge, query, BM25F, relaxation. | persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. | | `persia-blob` | Content-addressed chunks, dedup, garbage collection. | No API yet (M9) | | `persia` | The embedded API: `Db` and `Collection`. | Not implemented (M10). The crate exports no types. | | `persia-proto` | `persia.proto` and committed generated code. | Module docs only (M11) | | `persia-server` | gRPC and HTTP. Roles: writer, reader, router. | Placeholder: exits with “not implemented” (M11, M13) | | `persia-cli` | Sweep temporary and quarantine files. `inspect segment` and `dump segment` read one segment (`--verify` checks it). `compact`, `bench`, import, and search are later. | Sweep, inspect, dump, and `verify` exist. The rest is M10. | Dependencies point downward only (`persia-format` at the bottom). `cargo xtask check-deps` enforces the graph. The public facade orders blob writes before the document commit so the engine never has to know about blob bytes (SPEC §9.5). On a query, the design is: analyze the text, search each segment and the memtable, merge the per-segment top-k, and if there are too few hits run the relaxation cascade. The response includes which level matched. The compiler, the BM25F scorer, matching, and the top-k merge exist as library code. The cascade does not. See [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). # Code > What the Persia DB crates export today, with links to the source. The public Db API does not exist yet. The engine has a WAL, commit, segment merging, and a ranked search over a collection snapshot. This page quotes code that is on `main`. Persia DB is pre-alpha. The public persia crate has no Db::open. persia-engine can put, delete, and commit a document. persia-engine runs a ranked search over a collection snapshot: BM25F, filters, sorts, facets, and pages. `persia-format` is the on-disk codec crate. `persia-storage`, `persia-engine`, and `persia-analysis` also export library code, summarized below. `persia` does not. The `Db::open` sketch on the [embedded API](https://tomiok.github.io/persiaDB/guide/embedded-api/) page is illustrative: `persia::Db` is not in the tree. persia_engine::db::Database has open, create_collection, put, put_batch, delete, and commit. CollectionSnapshot::get fetches one document by id. CollectionViews::search runs a ranked search over a collection snapshot. There is no persia::Db, no search on the database handle, no put_struct, no put_blob, no get_blob, and no OpenOptions. Those SPEC names are not the public embedded API. ## Crate layout [Section titled “Crate layout”](#crate-layout) The workspace members are the directories under `crates/` plus `xtask` ([root Cargo.toml](https://github.com/tomiok/persiaDB/blob/main/Cargo.toml)). What each file says about itself: | Crate | Source | What the file says | In the tree | | ----------------- | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `persia-format` | [lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/lib.rs) | On-disk primitives: encodings, frames, superblock, checksums. No I/O policy. | Encodings, checksums, block codecs and framing, the container header, the superblock, frames (including the streaming encoder), mmap, `BlobId`, and `BlobRef`. | | `persia` | [lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia/src/lib.rs) | The public embedded API (`Db`, `Collection`). | The module docs only. No `Db` type. | | `persia-engine` | [lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-engine/src/lib.rs) | Segments, indexing, WAL, query, scoring, relaxation. Synchronous. | persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. Public modules: db, document, manifest, memtable, merge, query, schema, score, search, segment, stats, tombstone, view, and wal. | | `persia-storage` | [lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-storage/src/lib.rs) | `Storage` trait and backends (SPEC §10). | The trait, memory, local file, and the single-file container. No S3, GCS, or Azure. | | `persia-analysis` | [lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-analysis/src/lib.rs) | Tokenizers, normalizers, stemmers (SPEC §6.3). | Tokenizers, filters, `en` / `es` / `pt` / `fr` / `de` / `it` stemmers, and built-in analyzers. | | `persia-blob` | [lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-blob/src/lib.rs) | Content-addressed chunks, dedup, GC (SPEC §9). | Module docs only. | `persia-server` is a placeholder binary that prints “not implemented (M11)” and exits with status 1. `persia-cli` lists and sweeps temporary and quarantine files, and it inspects and dumps one segment (`persia inspect segment`, `persia dump segment`, with `--verify` on inspect), and verifies a whole database (`persia verify`). It does not search. `persia-proto` is a placeholder library for M11. The public modules and re-exports of `persia-format` are the list in [lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/lib.rs): ```rust pub mod align; pub mod bitpack; pub mod blob; pub mod block; pub mod checksum; pub mod checksummed; pub mod compress; pub mod delta; pub mod error; pub mod for_codec; pub mod frame; pub mod hash; pub mod header; pub mod mmap; pub mod offset; pub mod postings; pub mod reader; pub mod source; pub mod superblock; pub mod varint; pub mod writer; pub mod zigzag; pub use align::ALIGNMENT; pub use blob::{BlobId, BlobRef}; pub use error::{Corruption, EncodeError, Error, Result, SuperblockField, Unsupported}; pub use offset::{Len, Offset}; pub use reader::Reader; pub use source::ByteSource; pub use writer::Writer; ``` Integer codecs (`delta`, `bitpack`, `postings`, `for_codec`), varints, zigzag, and the bounds-checked `Reader` / `Writer` live in those modules. They are library primitives. Segment writers in `persia-engine` use them (M5). A ranked search over a collection snapshot is library code in `persia-engine` (`CollectionViews::search`). The query compiler, the filter DSL, and the BM25F scorer are (ROADMAP 7.1–7.3). See [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). ## Checksums [Section titled “Checksums”](#checksums) CRC32C is the framing checksum (SPEC §4.1). The wrapper is [checksum.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/checksum.rs). One-shot and streaming: ```rust pub fn crc32c(data: &[u8]) -> u32 { crc32c::crc32c(data) } ``` The streaming hasher is the same file, through `finish` (lines 51–70; `verify` follows): ```rust #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] pub struct Crc32c { state: u32, } impl Crc32c { /// A checksum of no bytes. pub const fn new() -> Self { Self { state: 0 } } /// Appends `data`. pub fn update(&mut self, data: &[u8]) { self.state = crc32c::crc32c_append(self.state, data); } /// The checksum of everything appended so far. Does not reset: more data can still be appended. pub const fn finish(self) -> u32 { self.state } ``` A mismatch is `Error::Corrupt` with `Corruption::ChecksumMismatch`, not a panic ([verify_crc32c](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/checksum.rs#L23-L25)). xxh3-64 and xxh3-128, seed 0, are the content and routing hashes. They are a different tool from CRC32C, and blob ids are SHA-256 on purpose ([hash.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/hash.rs), [ADR-0002](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0002-blob-hash-sha256.md)): ```rust pub fn xxh3_64(data: &[u8]) -> u64 { xxhash_rust::xxh3::xxh3_64(data) } ``` ## Checksummed\ [Section titled “Checksummed\”](#checksummedt) Fixed-size structures whose CRC sits inside the region use [`Checksummed`](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/checksummed.rs#L98-L99). The container header and the superblock slots use it (SPEC §4.2, §4.3; `header.rs`, `superblock.rs`). A type describes where the CRC field sits: ```rust pub trait ChecksummedLayout: Sized { /// Total size of the region in bytes, CRC field included. const LEN: usize; /// Position of the `u32` CRC32C field within the region. const CRC_OFFSET: usize; fn encode_fields(&self, w: &mut Writer); fn decode_fields(r: &mut Reader<'_>) -> Result; } ``` ```rust pub struct Checksummed(pub T); ``` `encode` writes the fields, checksums every byte except the CRC field, and fills that field in. `decode` checks the CRC before it interprets the fields ([checksummed.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/checksummed.rs#L42-L168)). Frames are variable-length and do not use this helper. Encode, decode, and scan live in `frame.rs`. ## Block codecs [Section titled “Block codecs”](#block-codecs) [compress.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/compress.rs) defines the codec byte from SPEC §5.2 and [ADR-0005](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0005-block-compression-codecs.md): ```rust #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] #[repr(u8)] pub enum Codec { /// Stored as is. None = 0, /// One zstd frame that declares its content size. Zstd = 1, } ``` Any other byte is `Error::Corrupt` with `Corruption::UnknownCodec`. ```rust pub fn compress_block( codec: Codec, level: i32, src: &[u8], out: &mut Vec, ) -> Result { match codec { Codec::None => { out.extend_from_slice(src); Ok(src.len()) } Codec::Zstd => compress_zstd(level, src, out), } } ``` Decompression takes the declared length and a caller-supplied maximum, and rejects a larger length before it allocates ([decompress_block](https://github.com/tomiok/persiaDB/blob/main/crates/persia-format/src/compress.rs#L146-L153)): ```rust pub fn decompress_block( codec: Codec, src: &[u8], uncompressed_len: u64, max_len: usize, offset: u64, out: &mut Vec, ) -> Result<()> { ``` A zstd payload has to be exactly one frame whose declared content size equals `uncompressed_len`. The on-disk block header, the block table, and random-access reads live in `block.rs` (roadmap 1.5). An allocation failure while decoding is `Error::OutOfMemory`, not corruption (roadmap 1.4.4). ## What is not here [Section titled “What is not here”](#what-is-not-here) Opening `app.persia`, putting a document, and searching are the embedded API in the design. The crate that will own them currently contains this, and nothing else ([crates/persia/src/lib.rs](https://github.com/tomiok/persiaDB/blob/main/crates/persia/src/lib.rs)): ```rust //! Persia DB: an embedded, single-file, cloud-native search database. //! //! This is the public embedded API (`Db`, `Collection`). #![forbid(unsafe_code)] ``` # Persia DB vs SQLite FTS5, Tantivy, and Meilisearch > Persia DB versus SQLite FTS5, Tantivy, and Meilisearch: embedded or server, single file or segments, language, and maturity. Pre-alpha; no persia::Db search API yet. > **Status: pre-alpha.** Persia DB is pre-alpha. The public persia crate has no Db::open. persia-engine can put, delete, and commit a document. persia-engine runs a ranked search over a collection snapshot: BM25F, filters, sorts, facets, and pages. `persia::Db::open` and `search` elsewhere on this site are illustrative, from [SPEC.md](https://github.com/tomiok/persiaDB/blob/main/SPEC.md). Live status is [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md). This page places the Persia DB design next to [SQLite FTS5](https://sqlite.org/fts5.html), [Tantivy](https://github.com/quickwit-oss/tantivy), and [Meilisearch](https://www.meilisearch.com/docs). It is not a benchmark. It does not say that any of them is faster. Facts about those projects are the stable ones in their own documentation. Anything Persia DB does not implement yet is marked planned. When this page and the SPEC disagree, the SPEC wins. ## At a glance [Section titled “At a glance”](#at-a-glance) | | Persia DB | SQLite FTS5 | Tantivy | Meilisearch | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------ | | What it is | Embedded search database, written in Rust. An optional server is planned. | Full-text virtual-table module inside SQLite, a C library. | Full-text search library, written in Rust. Its docs describe it as Lucene, in Rust. | Standalone search server, written in Rust. | | How you run it | Planned: in-process. `Db::open` on a file. No daemon for the library. | In-process. Link SQLite and query with SQL. | In-process. Link the crate. Indexing and serving are your code. | A separate process. Clients use an HTTP API. | | What you store | Planned: one `.persia` file, plus an optional `.persia.blobs` pack. | The FTS5 index lives in the SQLite database file. | A directory of segment files. | The server’s own data directory. | | How you query | Planned: a collection API, BM25F, filters, and facets. | SQL `MATCH`, with a `bm25()` ranking function. | Queries against a schema declared in code. | JSON over HTTP. Typo tolerance is on by default. | | Object storage | Planned: the same segment bytes locally or in S3, GCS, or Azure Blob. | A local database file. | A directory the library writes. | You run the server. | | Today | Persia DB is pre-alpha. The public persia crate has no Db::open. persia-engine can put, delete, and commit a document. persia-engine runs a ranked search over a collection snapshot: BM25F, filters, sorts, facets, and pages. | Part of SQLite. | A released library. | A released server. | ## Persia DB [Section titled “Persia DB”](#persia-db) The intended product is an embedded, single-file, cloud-native search database. Think “SQLite for search”: open a file, index documents, and get ranked results that, by default, never come back empty. That is the design in [SPEC §1](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#1-goals-and-non-goals) and on the [overview](https://tomiok.github.io/persiaDB/guide/overview/). It is not what the crate does today. [What is an embedded search database?](https://tomiok.github.io/persiaDB/guide/embedded-search/) defines that in-process, single-file shape, and separates it from running a search server. persia-format implements the on-disk codecs (M1 done, 53/53). persia-storage implements the Storage trait, memory and local-file backends, and the single-file container (M2 done, 46/46). S3, GCS, and Azure have not been started (M12). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. persia-analysis implements tokenizers and the built-in analyzers de, en, es, fr, it, keyword, pt, standard, and whitespace (M4 done, 29/29). It does not search. persia-blob exports no API (M9 not started; its dependencies are done). The public persia crate exports no Db and no Collection. The crates are unpublished (publish = false, version 0.0.0). Building one segment is not `persia::Db::open`. persia_engine::db::Database has open, create_collection, put, put_batch, delete, and commit. CollectionSnapshot::get fetches one document by id. CollectionViews::search runs a ranked search over a collection snapshot. There is no persia::Db, no search on the database handle, no put_struct, no put_blob, no get_blob, and no OpenOptions. Those SPEC names are not the public embedded API. The [getting started](https://tomiok.github.io/persiaDB/guide/getting-started/) page separates that library code from the illustrative API. The first release a caller can use is `v0.1` (roadmap M10). The format stays unstable until `v1.0` (M15). Counts on the [roadmap](https://tomiok.github.io/persiaDB/roadmap/) are read from PROGRESS.md when the site is built. Specified, and searchable only as engine library code today: * The query compiler, the filter DSL, and a BM25F scorer are library code (ROADMAP 7.1–7.3). Library code (M7 in progress, 46/51): matching, sorts, pages, and facets. There is no `search` call on a database handle (M10). Analysis libraries are in the tree. The project does not wrap Tantivy, Lucene, or SQLite ([ADR-0001](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0001-own-engine-and-allowed-dependencies.md)). * A query with too few hits is relaxed step by step, and the response says what changed (SPEC §7.6). That cascade is not implemented. See [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/) and [Features](https://tomiok.github.io/persiaDB/features/). * Segments are immutable. A local commit and an object-storage commit publish the same segment bytes. Query nodes are stateless. See [How it works](https://tomiok.github.io/persiaDB/architecture/) and [File format and storage](https://tomiok.github.io/persiaDB/guide/file-format/). * A local database is two files: `.persia` and `.persia.blobs` (SPEC §4, §9). * Out of scope for v1: SQL, joins, vector search, more than one writer on a shard, and encryption inside the file (SPEC §1.2). ## SQLite FTS5 [Section titled “SQLite FTS5”](#sqlite-fts5) [FTS5](https://sqlite.org/fts5.html) is an SQLite virtual table module for full-text search. You create a table with `CREATE VIRTUAL TABLE … USING fts5(…)` and search it with SQL `MATCH`. The built-in `bm25()` function ranks matches. The index is stored in that same SQLite database file: FTS5 keeps it in shadow tables, which applications do not query directly. SQLite is a C library. The database and the full-text index run in your process. There is no search server to deploy. The rest of the application’s rows can live in the same file. FTS5 is not an object-storage index, and it is not a Rust API. The Persia DB design aims at the same “open a file” step, then specifies a search database rather than a SQL engine: no SQL, immutable segments, and a planned path that stores those segment bytes in object storage. That search path is not implemented. ## Tantivy [Section titled “Tantivy”](#tantivy) [Tantivy](https://github.com/quickwit-oss/tantivy) is a Rust library for full-text search. The project describes itself as Lucene, in Rust. You declare a schema, create an index in a directory, add documents, commit, and run queries you compose. The index is a directory of segments, not one database file. An HTTP API or a cluster is something you build, or another project; Tantivy itself is the library. The language and the embedded shape are the nearest match to Persia DB. The design then diverges. The local container is one `.persia` file (SPEC §4). The same segment bytes are specified for object storage (SPEC §10). Relaxation is specified to be on by default (SPEC §7.6). The engine is not Tantivy: ADR-0001 forbids depending on another search engine. Tantivy is a library you can use today. Persia DB is not. ## Meilisearch [Section titled “Meilisearch”](#meilisearch) [Meilisearch](https://www.meilisearch.com/docs) is a search server, written in Rust. You run it as its own process and send documents and queries over HTTP. Typo tolerance is on unless you disable it. Filtering, faceted search, and ranking rules are part of that server. Data lives in the server’s directory, not in a file the application opens. The shared idea is a search API that tolerates a small mistake in the query, and an implementation in Rust. The mechanisms differ. Meilisearch’s default is typo tolerance on the terms it searches. Persia DB’s planned default is a cascade: synonyms, dropped terms, prefix, a fuzzy step, then a fallback, with the response naming the step (SPEC §7.6). That cascade is not implemented. [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/) is the longer account of that design, and of the compiler and scorer that do exist. The deployment is different as well. Meilisearch is a service you operate. Persia DB is specified as a library you open on a file, with a server only as a later option ([Server, cloud, and Kubernetes](https://tomiok.github.io/persiaDB/guide/server/), M11). Meilisearch is a released server. The Persia DB server does not exist. ## Lucene and Elasticsearch [Section titled “Lucene and Elasticsearch”](#lucene-and-elasticsearch) [Apache Lucene](https://lucene.apache.org/) is the Java library Tantivy is modeled on: an embedded index of segments in a directory. [Elasticsearch](https://www.elastic.co/elasticsearch) is a distributed server built on Lucene. Running it means operating a cluster, which is a different job from linking a library or opening one file. The planned Persia DB server would embed the same engine as the library, and segment bytes would stay the same in object storage. That server is not written. ## When to pick which today [Section titled “When to pick which today”](#when-to-pick-which-today) For a search feature in production today, use one of the established tools. Persia DB has no `persia::Db` search API yet (M10); the engine library code can search. * **SQLite FTS5**, when the data is already in SQLite, or you want SQL and full-text in one database file and one process. * **Tantivy**, when you are writing Rust, you want a library, and you will own the schema, the index directory, and how results are served. * **Meilisearch**, when you want a separate search service, an HTTP API, and typo tolerance without building the engine. * **Persia DB**, when you are reading the design or following the [roadmap](https://tomiok.github.io/persiaDB/roadmap/). Not when an application needs search. The combination in the SPEC — an embedded Rust API, one local file, the same segment bytes in object storage, and relaxation on by default — is specified. It is not shipped. ## Read next [Section titled “Read next”](#read-next) * [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/) — query relaxation, fuzzy matching, BM25F, and filters, marked designed or implemented. * [What is an embedded search database?](https://tomiok.github.io/persiaDB/guide/embedded-search/) — in-process full-text search, one file, and no server, and what this repository implements. * [Overview](https://tomiok.github.io/persiaDB/guide/overview/) — what the project is, and what already exists. * [FAQ](https://tomiok.github.io/persiaDB/guide/faq/) — short answers on status, the file format, crash safety, SDKs, and the license. * [Getting started](https://tomiok.github.io/persiaDB/guide/getting-started/) — codecs you can call, and the API that is still illustrative. * [Features](https://tomiok.github.io/persiaDB/features/) and the [feature reference](https://tomiok.github.io/persiaDB/guide/features/) — each capability marked planned or done. * [How it works](https://tomiok.github.io/persiaDB/architecture/) and [file format and storage](https://tomiok.github.io/persiaDB/guide/file-format/) — segments, frames, and backends. * [Roadmap](https://tomiok.github.io/persiaDB/roadmap/) — milestone status. The live counter is PROGRESS.md. # Key concepts > The names Persia uses for a database, a segment, a commit, and relaxation. Names below are the ones used in code and in the SPEC. The short definitions are [SPEC §2](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#2-terminology). | Term | Meaning | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Database** | One Persia DB: collections plus blobs. Opened by a filesystem path or a URL. | | **Collection** | Documents that share a schema, analyzers, and ranking config. | | **Document** | A record with a string `id`, typed fields, and an optional blob reference. | | **Segment** | An immutable, self-contained index for one collection (SPEC §5). Once written, it is never edited. | | **Memtable** | The in-memory buffer of recent writes. Searches see it, so a `put` is visible before the next flush. | | **WAL** | Write-ahead log of operations not yet flushed into a segment. | | **Manifest** | A versioned snapshot: live segments per collection, schemas, and the WAL position still needed. | | **Commit** | Atomic publication of a new manifest. Readers move from one manifest to the next; they never see a mix. | | **Frame** | One checksummed record in the local container (SPEC §4.4): WAL chunk, segment, manifest, padding, or tombstone sidecar. | | **Blob** | An opaque client file, addressed by the SHA-256 of its bytes and stored in chunks (SPEC §9, ADR-0002). | | **Shard** | An independent database that owns one hash range of document ids (SPEC §11). | | **Relaxation** | A controlled loosening of a query so the default experience is not an empty page (SPEC §7.6). The cascade is not implemented. See [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). | A few more words that show up in the roadmap and in contributor docs ([`docs/architecture.md`](https://github.com/tomiok/persiaDB/blob/main/docs/architecture.md)): | Term | Meaning | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Leaf** | One ROADMAP checkbox. One leaf is one issue and one pull request. | | **Milestone** | `M0` through `M15` are the MVP (v1.0). `M16` and later are post-MVP and name a target version. A milestone starts only after the milestones it depends on are done, unless the roadmap says otherwise. | | **ADR** | An architecture decision record in [`docs/adr/`](https://github.com/tomiok/persiaDB/blob/main/docs/adr/). The SPEC says what the system does; an ADR says why. | | **Relaxation level** | One step of the cascade, `L0` (exact) through `L5` (fallback). | Internal document ids are segment-local. Deletions never rewrite a segment: they live in a tombstone sidecar referenced by the manifest (SPEC §8.3). The exact sidecar bytes are still an ADR (ROADMAP 6.4.2, SPEC §18.3). # Contributing > How to pick a roadmap leaf, what a pull request has to pass, and where to report security issues. Persia DB is a database, so correctness comes first and speed comes after, with measurements. 1. Read [`SPEC.md`](https://github.com/tomiok/persiaDB/blob/main/SPEC.md) and [`CLAUDE.md`](https://github.com/tomiok/persiaDB/blob/main/CLAUDE.md). The rules apply to human and AI-assisted changes. 2. Pick a leaf from [`ROADMAP.md`](https://github.com/tomiok/persiaDB/blob/main/ROADMAP.md). [`PROGRESS.md`](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) lists what is ready next. One leaf, one pull request. 3. If the SPEC is unclear, or you want to change a decision, open an issue before writing the code. The code and the SPEC are not allowed to diverge silently. Setup is the pinned toolchain (rustup reads `rust-toolchain.toml`) and `cargo test --workspace`. Before a pull request: format, clippy with warnings denied, tests, `cargo xtask check-deps`, and `cargo xtask progress --check`. The long form is [`CONTRIBUTING.md`](https://github.com/tomiok/persiaDB/blob/main/CONTRIBUTING.md). On-disk format changes need all four of: a SPEC update, a version or flag bump, an ADR under [`docs/adr/`](https://github.com/tomiok/persiaDB/blob/main/docs/adr/), and a golden test. Bytes from disk, the network, or a user are never `unwrap`ped; corruption is `Err`. `unsafe` is confined to the memory-map module in `persia-format`, and any other use needs an ADR. Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/): `feat(engine): ...`, `fix(format): ...`, `docs: ...`. By participating you agree to the [Code of Conduct](https://github.com/tomiok/persiaDB/blob/main/CODE_OF_CONDUCT.md). Report security issues privately, as described in [`SECURITY.md`](https://github.com/tomiok/persiaDB/blob/main/SECURITY.md). The project is pre-alpha: only the latest commit on `main` receives fixes. Treat a panic or unbounded memory use on a crafted database file as a security issue (SPEC §17). # Embedded API > The planned Db and Collection API. Method names are illustrative and are not in the crate. **Planned.** The usage walkthrough, including what works today and the FAQ, is [Getting started](https://tomiok.github.io/persiaDB/guide/getting-started/). The public crate `persia` is a stub. The first release a caller is supposed to use is v0.1 (M10). That milestone is blocked on M6 (in progress), M7 (in progress), M8 (not started), and M9 (not started). persia_engine::db::Database has open, create_collection, put, put_batch, delete, and commit. CollectionSnapshot::get fetches one document by id. CollectionViews::search runs a ranked search over a collection snapshot. There is no persia::Db, no search on the database handle, no put_struct, no put_blob, no get_blob, and no OpenOptions. Those SPEC names are not the public embedded API. Method names below are illustrative and not final. This is the sketch already in the README: ```rust let db = persia::Db::open("app.persia")?; // Collections declare their schema up front (SPEC §6.2). let products = db.create_collection(schema)?; products.put(doc)?; products.commit()?; // No exact match for the typo? The query is relaxed (here: fuzzy, SPEC §7.6), // and the response says what changed: "runing" -> "running". let results = products.search("runing")?; ``` That `search` call is illustrative. Fuzzy matching is one level of the planned relaxation cascade, which is not implemented. Library code (M7 in progress, 46/51): the compiler, the filter DSL, and the scorer return ranked hits through CollectionViews::search. See [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). The intended surface (SPEC §8, §9.6, ROADMAP 10.1): * `Db::open` on a path or URL, `OpenOptions`, and `close`. * Create, open, and drop a collection, and read its schema back. * `put`, `put_batch`, `delete`, `get`, `commit`, and `search`. * Documents as a dynamic map, or as a `serde` struct (`put_struct`). * `put_blob` / `get_blob` / ranged reads. A convenience `put` that takes a document and a blob writes the blob first, then commits the document. * A stable error type. Library code returns `Result`; corrupted bytes are an error, never a panic. Handles are `Send` and `Sync`. Many readers share a snapshot. A second writer fails immediately. A `put` is visible to later searches on that database (read-your-writes, SPEC §8.2–§8.4). An optional `tokio` feature will offer a thin `spawn_blocking` wrapper (ROADMAP 10.1.6); the core API stays synchronous. Embedded mode has no authentication and no config file. Knobs, once they exist, are builder options with defaults: memtable size, commit interval, fsync policy, merge threads, search threads, cache sizes, block size, blob chunk size, and the `relax.*` settings (SPEC §14). The `persia` command-line tool’s later commands (`compact`, `stats`, JSONL import, a one-shot search, and a small bench) are part of the same milestone (ROADMAP 10.4) and are not in the binary. `persia inspect segment` and `persia dump segment` are, and `--verify` checks the segment. File-sweep commands exist too. None of that is `Db::open`. The `v0.1.0-alpha` tag is specified to say, in the release itself, that the format is still unstable (ROADMAP 10.5.2). # Feature reference > Reference for Persia DB search, BM25F ranking, the zero-results cascade, schema, durability, and blobs. Each item is planned or done. Ranked full-text search, filters, sorts, and facets run as library code in persia-engine, over a collection snapshot. There is no Db to open yet (M10). Relaxation is not started (M8), so a search that matches nothing returns no hits. The write path is in progress (M6, 49/58). Schema, analyzers, the container commit, segment bytes, and part of the write path also exist as library code. The public persia crate exports no Db and no Collection. A search runs only through persia-engine (CollectionViews::search). The crates are unpublished (publish = false, version 0.0.0). | Capability | Status | Defined in | | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | | Open a file and search | Not started (M10). Below it, persia_engine::db::Database opens a file and CollectionViews::search searches it. | SPEC §8, §10.2 | | Query AST, user-text compiler, filter DSL, limits, BM25F | Library code (M7 in progress, 46/51): the compiler, the filter DSL, and the scorer return ranked hits through CollectionViews::search. | SPEC §7.1, §7.2 | | Matching, filters, column sort, paging, facets | Library code (M7 in progress, 46/51): matching, sorts, pages, and facets. | SPEC §7.3–§7.5 | | Zero-results relaxation | Not started (M8). On by default in the design. | SPEC §7.6 | | Schema and documents | Done as library code (M3). Not the `Db` API. | SPEC §6 | | Analyzers (`standard`, `keyword`, `whitespace`, `en`, `es`, `pt`, `fr`, `de`, `it`) | Done as library code (M4). The query compiler runs them when it compiles user text. | SPEC §6.3 | | Segment writer and reader | Done (M5, 48/48). Segments are searched through CollectionViews::search. | SPEC §5 | | Crash-safe commit and recovery | Container commit is done (M2, 46/46). The document write path is in progress (M6, 49/58): WAL, flush, commit, and snapshots, deletes and upserts, and segment merging. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle. | SPEC §4.5, §8.6 | | Blob storage | Not started (M9; its dependencies are done). | SPEC §9 | | gRPC and HTTP server | Not started (M11). | SPEC §12 | | S3, GCS, Azure, replication | Not started (M12). | SPEC §10 | | Kubernetes, sharding | Not started (M13). | SPEC §11 | | Embedded Go and TypeScript bindings | Planned (after M10). Design only. | SPEC §13.1, [Embedded SDKs](https://tomiok.github.io/persiaDB/guide/sdks/) | | Rust and Go network SDKs | Not started (M14). | SPEC §13.2 | | Integer codecs and `None` / `Zstd` block codecs | Done (M1, 53/53). Library primitives. | SPEC §4.1, §5.2 | ## Search and ranking [Section titled “Search and ranking”](#search-and-ranking) A search takes user text, a field list, filters, a sort, a page, and optional facets (SPEC §7.1). The text is analyzed per field and compiled into a query: terms, phrases, prefixes, fuzzy clauses, and boolean must / should / must-not. The default operator is AND. Invalid syntax degrades to plain terms rather than failing the request (ROADMAP 7.1.2). Filters (term, range, exists, and / or / not) are separate from the text and are not relaxed unless you opt in. Ranking is BM25F (SPEC §7.2): each field has a weight, `k1`, and `b`, and a collection may add a static rank (for example `log1p` of a sales field). Collection statistics (`N`, document frequency, average field length) are summed across live segments. Scores must match the reference model within a documented epsilon (SPEC §7.2). `TermScorer` and `persia-engine/tests/reference` do that for one term (ROADMAP 7.3.4). Ranked hits over one segment or memtable view are library code (ROADMAP 7.4.1–7.4.4), and so is a collection search (`CollectionViews::search`). A search call on the database handle is not (M10). The longer account of typos, relaxation, and this scorer is [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). Over one view — a segment, or the memtable — `SearchPlan` collects the top k by score, then docid, with block-max WAND by default (`MaxScore` is the other strategy) (ROADMAP 7.4.1–7.4.4, SPEC §7.3). Pruning does not change the hits: pruned and exhaustive top k are the same documents, scores bit for bit. A segment view drops tombstoned documents in the postings iterator, before a document is scored or offered to the collector, so a deleted document cannot raise the top-k threshold. Block bounds may still cover deleted documents; that only prunes less (SPEC §7.3). Also library code: filter bitmaps (ROADMAP 7.4.5), the memtable hiding every older segment copy of an id (ROADMAP 7.4.6), searching segments in parallel (ROADMAP 7.6.1), sort by a column and a `search_after` cursor (ROADMAP 7.5.1, 7.5.2), facets (ROADMAP 7.5.3), and `explain: true` (ROADMAP 7.7.1), all through `CollectionViews::search`. Highlighting is post-v1. ## Zero-results relaxation [Section titled “Zero-results relaxation”](#zero-results-relaxation) Relaxation is on by default. If a query returns fewer hits than `min_results` (default 1), the engine walks a cascade and stops at the first level that produces enough hits (SPEC §7.6). With `fill` set, it keeps going until the page is full. Stricter levels always rank above looser ones. Filters stay strict unless `relax.filters` is `last_resort`. A time budget (default 50 ms) returns the best results so far and sets `truncated`. A query can turn the whole feature off with `relax: false`. | Level | Name | What it changes | | ----- | ------------ | -------------------------------------------------------------------------------------------------------- | | L0 | `exact` | Every analyzed term is required. | | L1 | `synonyms` | Terms expand through the collection’s synonym groups. | | L2 | `drop_terms` | Required-term ratio steps from 100% to 75%, 50%, then one term. Low-value terms go first. | | L3 | `prefix` | The last term (and other terms of length at least 3) expands through a prefix scan, capped (default 50). | | L4 | `fuzzy` | Edit distance 1 for terms of length 4–7, distance 2 for length 8 or more. Corrections are reported. | | L5 | `fallback` | A collection-defined last resort: top documents by static rank or recency, or nothing. | The response says what happened. This JSON is the shape in SPEC §7.6. It is illustrative: ```json "relaxation": { "level": "fuzzy", "applied": ["drop_terms:75%", "fuzzy"], "original": "zapatilas rojs", "effective": "zapatillas rojas", "corrected": { "zapatilas": "zapatillas", "rojs": "rojas" } } ``` Quality of this cascade is a product feature of the same milestone (ROADMAP M8): a synthetic typo suite, a zero-result rate before and after relaxation, and a regression gate. A tuning guide is planned as ROADMAP 8.4.4 (`docs/relaxation.md`). What is implemented today, and what is still only the design, is [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/). ## Schema, documents, and analysis [Section titled “Schema, documents, and analysis”](#schema-documents-and-analysis) A collection declares its schema up front (SPEC §6.2). Field types are `text`, `keyword`, `i64`, `f64`, `bool`, `datetime` (microseconds UTC), and `stored` (kept, not searchable). Every document has an `id` of at most 512 bytes, unique in the collection, and may reference one blob in v1 (several blobs per document is an open question, SPEC §18.2). Adding a field is allowed; existing documents read that field as missing. Changing a field’s type or analyzer is rejected and needs a new collection version (reindex). Unknown JSON fields are rejected when `strict` is true, and kept as stored-only fields when it is false (SPEC §6.4). Documents are submitted as JSON, or as typed structs in the SDKs, and stored as CBOR. The schema sketch in SPEC §6.2, illustrative: ```json { "name": "products", "fields": { "title": { "type": "text", "analyzer": "es", "boost": 3.0, "positions": true, "k1": 1.2, "b": 0.75 }, "body": { "type": "text", "analyzer": "es", "boost": 1.0 }, "brand": { "type": "keyword", "facet": true }, "price": { "type": "f64" }, "sales": { "type": "i64" } }, "ranking": { "static_rank": { "field": "sales", "weight": 0.1, "fn": "log1p" } }, "relaxation": { "enabled": true, "min_results": 1, "levels": "default", "fallback": "static_rank" }, "synonyms": [["tv", "television"]] } ``` Analyzers are a pipeline: character filters, then a tokenizer, then token filters (SPEC §6.3). Built-ins implemented (ROADMAP 4.4.2 and 4.4.10) are `standard` (Unicode word boundaries, NFKC, lowercase), `keyword`, `whitespace`, and language analyzers `en`, `es`, `pt`, `fr`, `de`, and `it` (stopwords, a Snowball-style stemmer, accent folding). Positions, used for phrases, are opt-in per field (SPEC §18.4). The analyzer’s name and version are stored in the schema so a later change is visible. Those pipelines live in `persia-analysis` (M4). Nothing here searches. ## Durability [Section titled “Durability”](#durability) Writes are `put` (upsert by id), `delete`, batches, `get`, and `commit`, plus an auto-commit policy (SPEC §8.1). The intended path (SPEC §8.2): 1. The operation is appended to the WAL. Durability of a call is `Buffered`, `Fsync`, or `Async`. The default at commit is `Fsync`. 2. The operation is applied to the memtable and becomes searchable. 3. When the memtable crosses a size or idle threshold (defaults 64 MiB and 10 seconds), it is flushed to a new immutable segment. 4. A commit publishes a new manifest. 5. WAL that is fully covered by committed segments is dropped. Local commit order is data frames, `fdatasync`, the manifest frame, `fdatasync`, then the inactive superblock slot, `fdatasync` (SPEC §4.5). A crash before the superblock write leaves the previous commit in place. Recovery reloads that manifest and replays the valid WAL prefix (SPEC §8.6). There is a single writer per database or shard (a file lock locally, a lease remotely). Any number of readers can be open; each reader sees one manifest snapshot (SPEC §8.4). Merges are tiered, run in the background, and publish through the same commit protocol (SPEC §8.5). Dead space in the local file is reclaimed by rewriting live data into a new file and atomically replacing the old one (SPEC §4.6). The numbered steps above are the path in SPEC §8.2. The container commit is done (M2, 46/46) and is crash-tested. The engine write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Database::open reloads the committed manifest, checks it against the superblock, and replays the WAL. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle. ## Blobs [Section titled “Blobs”](#blobs) The index stores a reference, never the file bytes (SPEC §9). A `blob_id` is the SHA-256 of the blob (ADR-0002). Persia hashes the bytes it receives; a caller-supplied id that does not match is an error. Chunks default to 4 MiB, each with a CRC32C, so uploads can resume and reads can be ranged. Blob bytes are stored as the client sent them (SPEC §9.2). The same content stored twice is one blob. Write order is deliberate: the blob is durable first, then the document that references it is committed (SPEC §9.5). A crash in between leaves an orphan, which is harmless. Garbage collection keeps anything a live manifest still references, and deletes orphans older than a grace period (default 24 hours). `verify` must fail if a referenced blob is missing. How chunks are staged before the hash is known is an open question (SPEC §18.8) and has to be decided before the pack-file work (ROADMAP 9.1.0). The `BlobId` and `BlobRef` value types exist in `persia-format` (ROADMAP 1.10.1): limits and binary encoding are in SPEC §9.1 and ADR-0011. # File format and storage > The local container, segments, block codecs, and the storage backends. High level. This page describes how the files are organized. Offsets, magic numbers, and CRC coverage are [SPEC §4](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#4-local-file-container) and [SPEC §5](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#5-segment-format). ## Local container [Section titled “Local container”](#local-container) `.persia` starts with a 12 KiB prefix: a 4 KiB header page (magic, format versions, compatibility flags, a database UUID) and two 4 KiB superblock pages. The slots are a double buffer on separate pages, so one bad sector cannot lose both. Open reads both, throws away a slot whose CRC fails, and uses the higher commit sequence (SPEC §4.2–§4.3). From byte 12288 the file is append-only frames. Each frame has a kind, a length, a payload padded to 8 bytes, and a CRC32C. A torn or corrupt frame ends recovery; the file is logically truncated there (SPEC §4.4). Integers are little-endian. Headers and frames use CRC32C. Internal content hashes and shard routing use xxh3 with seed 0. Blob ids are SHA-256 (ADR-0002), a content address that stays stable for the life of the blob. The compatibility rules (SPEC §15) are part of the design, and the bytes may still change before v1.0. An unknown major version or an unknown incompatible flag is a hard reject. An unknown read-only-compatible flag opens the file read-only. Every format change is a SPEC update, a version or flag bump, an ADR, and a golden fixture (see [Contributing](https://tomiok.github.io/persiaDB/guide/contributing/)). The container header, the superblock slots, and frames (encode, decode, scan, and the streaming encoder) are implemented in `persia-format` (ROADMAP 1.6 and 1.7). `persia-format` also has the encodings those structures use, plus the compression codecs below. ## Segments and blocks [Section titled “Segments and blocks”](#segments-and-blocks) A segment is a self-contained byte string: sections, then a footer with statistics, a section table, and a CRC (SPEC §5.1). It is a frame payload locally and a standalone object remotely, and the bytes match. Sections hold the term dictionary, postings, frequencies, skip lists, optional positions, length norms, columnar values, stored documents, the id map, and blob references (SPEC §5.3). Deleted ids are not a section. ``` ╔═ storage ════════════════════════════════════════════════════════╗ ║ local file object storage ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ header, 12 KiB ┃ same bytes ┃ manifests/seq ┃ ║░ ║ ┃ two superblocks ┃ ┃ segments/... ┃ ║░ ║ ┃ append-only frames ┃ ┃ blobs/... ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ sections, then sections ┃ ║░ ║ ┃ footer: stats, table, CRC ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ 16 KiB blocks · codec None or Zstd ║░ ╚══════════════════════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ``` The same segment bytes in a local frame and in object storage ([SPEC §5](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#5-segment-format), [§10](https://github.com/tomiok/persiaDB/blob/main/SPEC.md#10-storage-abstraction-and-cloud-backends)). Deleted ids are not a section. Block framing is done (M1, 53/53). Memory and local-file backends are done (M2, 46/46). S3, GCS, and Azure are not started (M12). Large sections are fixed-size blocks, 16 KiB uncompressed by default. Each block carries its own CRC, codec, and uncompressed length, so object storage can fetch one block with a range request (SPEC §5.2). Codecs: | Codec byte | Name | Meaning | | ---------- | ------ | ----------------------------------------------------------------------------- | | 0 | `None` | The payload is the raw bytes. Its length equals `uncompressed_len`. | | 1 | `Zstd` | Exactly one zstd frame whose declared content size equals `uncompressed_len`. | Any other codec byte is corruption. The reader refuses a declared length above the caller’s maximum before it allocates (the decompression-bomb guard, ADR-0005). Writers may store `None` when zstd does not shrink the block. The zstd level is a writer choice and is not stored. `compress` / `decompress` for those two codecs is done (ROADMAP 1.4). The on-disk block header, the block table, and random-access reads are implemented (ROADMAP 1.5, SPEC §5.2). Integer codecs that sit inside blocks (delta-coded doc ids, bit-packing, postings blocks, frame-of-reference for numbers) are implemented and tested (ROADMAP 1.2, ADR-0004). Segment section writers in `persia-engine` use them. persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. The leaf count is on the [roadmap](https://tomiok.github.io/persiaDB/roadmap/). ## One storage interface, several backends [Section titled “One storage interface, several backends”](#one-storage-interface-several-backends) The engine is specified to see every backend as the same trait: get, ranged get, put, put-if-absent, list, delete (SPEC §10.1). The `Storage` trait and the memory and local-file backends are implemented (M2). S3, GCS, and Azure Blob are not (M12). The write path already uses the trait. Query execution that would search through it is not written. | Backend | How a database is addressed | Milestone | | -------------- | ------------------------------- | --------- | | Local file | `app.persia`, or `file:///…` | M2 | | Memory | `memory://` | M2 | | S3 (and MinIO) | `s3://bucket/prefix` | M12 | | GCS | `gs://bucket/prefix` | M12 | | Azure Blob | `az://account/container/prefix` | M12 | Cloud credentials use each provider’s normal chain (environment, config files, instance metadata). Persia-specific configuration is an override, not a requirement. An endpoint override exists for the development emulators (MinIO, fake-gcs-server, Azurite). A remote commit uploads new segments, then creates `manifests/` with put-if-absent (SPEC §10.4). Conflict means another writer won. Readers discover the newest manifest by listing, with an optional `CURRENT` hint. Remote reads go through a block cache (memory, and optionally local disk) keyed by object and block (SPEC §10.6). `replicate` mode keeps the local file as the primary and uploads committed manifests in order, without blocking local writes on the network (SPEC §10.5). Remote readers of that replica are eventually consistent; lag is a metric. # License > Persia DB is licensed under the Apache License, Version 2.0, including copyright, NOTICE attribution, and the terms for contributions. Persia DB is licensed under the [Apache License, Version 2.0](https://github.com/tomiok/persiaDB/blob/main/LICENSE). The repository has one `LICENSE` file, and that file is the Apache License, Version 2.0. The same text is published at [apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0). Copyright and attribution live in [`NOTICE`](https://github.com/tomiok/persiaDB/blob/main/NOTICE): > Persia DB Copyright 2026 The Persia DB Authors > > This product includes software developed by The Persia DB Authors. Licensed under the Apache License, Version 2.0 (see LICENSE). The workspace records the same license in the root `Cargo.toml` (`license = "Apache-2.0"`). A short answer is also on the [FAQ](https://tomiok.github.io/persiaDB/guide/faq/). This page describes what that license says. It is not legal advice. ## Embedding in an application [Section titled “Embedding in an application”](#embedding-in-an-application) The license grants a copyright license (section 2) and a patent license (section 3). Both are perpetual, worldwide, non-exclusive, no-charge, and royalty-free. It does not split commercial use from other use, and it does not ask for a separate paid license to embed the library. The same grant is what an open-source application and a commercial application are both reading. The copyright grant covers reproducing the work, preparing derivative works of it, and distributing it in source form or object form. That is the clause that covers shipping Persia DB inside another program. If you redistribute Persia DB, or a derivative of it, section 4 requires all of the following: * Give recipients a copy of the Apache License, Version 2.0. * State, in any file you changed, that you changed it. * Keep the copyright, patent, trademark, and attribution notices from the source, except notices that do not apply to the part you kept. * Keep the attribution notices from `NOTICE`. The license lists where that copy can live: a NOTICE file shipped with the derivative, the source or documentation, or a display where such third-party notices normally appear. Section 1 defines a derivative work. It says that definition does not include a work that remains separable from Persia DB, or that merely links (or binds by name) to its interfaces. An application that calls the library and keeps its own code separate is, under that sentence, not itself a derivative work. The license still applies to the Persia DB code distributed with that application: recipients of that code need the license, the notices, and `NOTICE`. Section 4 also says you may add your own copyright statement to your modifications, and you may apply additional or different license terms to your modifications, or to a derivative work as a whole, provided your use of the original work still meets the conditions above. The patent grant ends for the party who files patent litigation claiming that the work, or a contribution incorporated in it, constitutes patent infringement (section 3). Section 6 does not grant trademark rights in the project’s names, except the reasonable and customary use needed to describe the origin of the work and to reproduce `NOTICE`. Sections 7 and 8 state that the work is provided as is, without warranty, and they limit contributor liability. Section 7 also says you are responsible for deciding whether using or redistributing the work is appropriate. ## Contributions [Section titled “Contributions”](#contributions) Unless a contributor explicitly states otherwise, a contribution intentionally submitted for inclusion in Persia DB is under this same license, without additional terms (section 5). That sentence does not replace a separate license agreement already made with the project. How to send a change is on the [contributing](https://tomiok.github.io/persiaDB/guide/contributing/) page. By participating you also agree to the [Code of Conduct](https://github.com/tomiok/persiaDB/blob/main/CODE_OF_CONDUCT.md). ## Other licenses in the tree [Section titled “Other licenses in the tree”](#other-licenses-in-the-tree) `LICENSE` is the license for Persia DB. Third-party crates keep their own licenses. [`deny.toml`](https://github.com/tomiok/persiaDB/blob/main/deny.toml) allows a set of permissive licenses for that dependency graph, including Apache-2.0 and MIT, and `cargo deny check` enforces the list. Those terms belong to the dependencies. They are not a second license on Persia DB. The fuzz-only `libfuzzer-sys` crate is an exception in that file (NCSA) and is not part of the database. # Milestones and dependencies > What each Persia DB milestone from M0 to v1.0 delivers, which milestones it depends on, and the post-MVP milestones through v3.0. Work is a tree: milestone, then section, then leaf. Every leaf is one issue and one pull request. Status lives on the leaves only (`[ ]` todo, `[~]` in progress, `[x]` done). [`PROGRESS.md`](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) derives the rest, and it is the live status. The table below is that file, read when the site is built: 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. A milestone is **done** when every leaf is done, **in progress** when some leaves are done, **ready** when its dependencies are done and none of its own leaves have started, and **blocked** while a dependency is unfinished. The paragraphs under each heading say what the milestone is for. The Snapshot column is the status. | Milestone | Delivers | Depends on | Snapshot | | ------------------------------------------- | ---------------------------------------- | ----------------------------------------------------------- | ------------------ | | **M0** Project bootstrap | Workspace, CI, tooling, docs scaffolding | — | In progress, 35/36 | | **M1** Format primitives | Encodings, codecs, header, frames | M0 | Done, 53/53 | | **M2** Storage abstraction & local backends | Storage, memory, local file, container | M1 | Done, 46/46 | | **M3** Schema, documents, collections | Collections, documents, manifests | M1 | Done, 15/15 | | **M4** Text analysis | Tokenizers, stemmers, analyzers | M0 | Done, 29/29 | | **M5** Segment writer & reader | Segment writer and reader | M1, M3, M4 | Done, 48/48 | | **M6** Write path, durability, merging | WAL, commit, merge, recovery | M2, M5 | In progress, 49/58 | | **M7** Query engine & ranking | Parser, BM25F, top-k, facets | M5 | In progress, 46/51 | | **M8** Zero-results avoidance | The zero-results cascade | M7 | Blocked, 0/29 | | **M9** Blob store | Pack file, remote blobs, GC | M1, M2 | Ready, 0/20 | | **M10** Embedded API | Db and Collection, CLI, v0.1 | M6, M7, M8, M9 | Blocked, 0/19 | | **M11** Server | gRPC, HTTP, image | M10 | Blocked, 0/23 | | **M12** Cloud storage backends | S3, GCS, Azure, cache, replicate | M2, M6, M9, M11 | Blocked, 0/29 | | **M13** Distribution & Kubernetes | Lease, shards, router, Helm | M11, M12 | Blocked, 0/24 | | **M14** SDKs | Rust and Go network clients | M11, M13 | Blocked, 0/19 | | **M15** Benchmarks, hardening, v1.0 | Benchmarks, hardening, format freeze | M1, M2, M3, M4, M5, M6, M7, M8, M9, M10, M11, M12, M13, M14 | Blocked, 0/22 | The generated ready-next list is in [`PROGRESS.md`](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md). That percentage is M0–M15 only. Milestones after v1.0 are at the end of this page. ## M0 — Project bootstrap [Section titled “M0 — Project bootstrap”](#m0--project-bootstrap) Done. The Cargo workspace and its crates, the pinned toolchain, CI (fmt, clippy, tests, `cargo deny`, Miri on `persia-format` and the pure parsers of `persia-engine`), the `justfile`, repo tooling in Rust (`cargo xtask`, ADR-0003), issue templates, and the documentation scaffolding: this repository’s rules, the ADR process, ADR-0001 and ADR-0002, and [`docs/architecture.md`](https://github.com/tomiok/persiaDB/blob/main/docs/architecture.md). No dependencies. ## M1 — Format primitives [Section titled “M1 — Format primitives”](#m1--format-primitives) Done. Depends on M0. Pure functions in `persia-format`, no I/O, which is why the bar for corruption handling is high: hostile bytes return an error. It covers little-endian cursors, varints, zigzag, checked offsets, integer codecs, CRC32C, xxh3, checksummed structs, the `None` / `Zstd` codecs, block framing, the container header and double-buffered superblock, frames (including the streaming encoder), the memory-map reader, golden container fixtures, and `BlobId` / `BlobRef`. The leaf list is [`ROADMAP.md`](https://github.com/tomiok/persiaDB/blob/main/ROADMAP.md). ## M2 — Storage abstraction and local backends [Section titled “M2 — Storage abstraction and local backends”](#m2--storage-abstraction-and-local-backends) Done. Depends on M1. The `Storage` trait and its error types, an in-memory backend, a deterministic failure injector for crash tests, a local object-per-file backend, and the single-file container adapter: create, append, the three-step commit, recovery, and compaction, each with crash tests. Also the first docker-compose file for later cloud emulators. ## M3 — Schema, documents, collections [Section titled “M3 — Schema, documents, collections”](#m3--schema-documents-collections) Done. Depends on M1. Field types and collection schema (including a stable schema hash and evolution rules), document validation, JSON and CBOR, and the manifest model. Segment writing (M5) needs this and M4. ## M4 — Text analysis [Section titled “M4 — Text analysis”](#m4--text-analysis) Done. Depends on M0. `persia-analysis` implements Unicode and whitespace tokenizers, NFKC, lowercasing, accent folding, English, Spanish, Portuguese, French, German, and Italian stopword lists and stemmers, synonyms, edge n-grams, and the named analyzer pipelines (`standard`, `keyword`, `whitespace`, `en`, `es`, `pt`, `fr`, `de`, `it`). Segment writing (M5) needs this milestone. Its state is the table above. ## M5 — Segment writer and reader [Section titled “M5 — Segment writer and reader”](#m5--segment-writer-and-reader) Depends on M1, M3, and M4. Its state is the table above. The in-memory index builder and every section writer and reader: term dictionary, postings, positions, norms, columns, stored documents, ids, blob references, and the footer. Includes `verify`, inspect and dump commands, property tests against a model, golden segments, and fuzz targets. This is the heart of the index format. Query (M7) can start from a reader; the full write path (M6) also needs storage. ## M6 — Write path, durability, merging [Section titled “M6 — Write path, durability, merging”](#m6--write-path-durability-merging) Depends on M2 and M5. The memtable, the WAL and its replay, flush and the local commit protocol, upserts and deletes via tombstone sidecars, tiered compaction, and open/recovery. Crash-loop tests (kill the process and reopen) land here. The embedded API waits on this milestone. Its state is the table above. The container commit is done (M2, 46/46) and is crash-tested. The engine write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Database::open reloads the committed manifest, checks it against the superblock, and replays the WAL. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle. ## M7 — Query engine and ranking [Section titled “M7 — Query engine and ranking”](#m7--query-engine-and-ranking) Depends on M5. End-to-end tests also need M6. The query AST and a parser that does not fail on messy user text, collection statistics, BM25F checked against hand-computed examples and the reference scorer, boolean and phrase matching, top-k with block-max WAND, filters, sort, pagination, facets, parallel segment search, and explain. Benchmarks of this milestone may start the v1 performance work early (M15). In the tree today: the query AST and compiler, the filter DSL, query limits, collection statistics, the BM25F scorer (ROADMAP 7.1–7.3), and matching with score top-k over one view, including block-max WAND (ROADMAP 7.4.1–7.4.4). Filter evaluation, facets, sort, parallel segment search, and explain are in the tree too (ROADMAP 7.4.5, 7.5, 7.6, 7.7), through `CollectionViews::search`. [Typo-tolerant search](https://tomiok.github.io/persiaDB/guide/typo-tolerant-search/) is that split. The relaxation milestone below has not been started. ## M8 — Zero-results avoidance [Section titled “M8 — Zero-results avoidance”](#m8--zero-results-avoidance) Depends on M7. The cascade in [Feature reference](https://tomiok.github.io/persiaDB/guide/features/): orchestration, per-level score offsets, the time budget, synonyms, dropping terms, prefix, fuzzy (Levenshtein automaton over the term dictionary), the fallback, and the response metadata. Plus a synthetic typo suite and a regression gate on the zero-result rate. The embedded `v0.1` release depends on this milestone (M10 depends on M8). ## M9 — Blob store [Section titled “M9 — Blob store”](#m9--blob-store) Depends on M1 and M2. An ADR for upload staging first (SPEC §18.8), then hashing and chunking, the local pack file, the remote key layout, ordering a blob write before the document commit, and garbage collection that never deletes a blob a live snapshot still references. ## M10 — Embedded API [Section titled “M10 — Embedded API”](#m10--embedded-api) Depends on M6, M7, M8, and M9. The first user-facing release. Public `Db` and `Collection`, concurrency and shutdown semantics, rustdoc and examples, a README quickstart that CI actually runs, the `persia` CLI, and a `v0.1.0-alpha` tag that states the format is unstable. See [Embedded API](https://tomiok.github.io/persiaDB/guide/embedded-api/). ## M11 — Server [Section titled “M11 — Server”](#m11--server) Depends on M10. `persia.proto`, committed generated code, the gRPC service, the HTTP/JSON gateway and its OpenAPI spec, configuration, health checks, metrics and tracing, authentication and TLS, and the container image. See [Server, cloud, and Kubernetes](https://tomiok.github.io/persiaDB/guide/server/). ## M12 — Cloud storage backends [Section titled “M12 — Cloud storage backends”](#m12--cloud-storage-backends) Depends on M2, M6, M9, and M11. S3, GCS, and Azure through a shared async layer, the remote commit protocol, a block cache, replicated mode, and conformance tests against MinIO, fake-gcs-server, and Azurite. Documentation of IAM and request cost is part of the milestone. ## M13 — Distribution and Kubernetes [Section titled “M13 — Distribution and Kubernetes”](#m13--distribution-and-kubernetes) Depends on M11 and M12. Writer leases and fencing, reader refresh, hash sharding and the router, role flags and discovery, the Helm chart, and a `kind` end-to-end test (ingest, query, kill the writer, fail over, roll). Resharding is a design record only for v1, with an optional prototype behind a flag. ## M14 — SDKs [Section titled “M14 — SDKs”](#m14--sdks) Depends on M11 for a stable protocol, and on M13 for shard-aware routing. A language-neutral conformance suite, then the Rust client and the Go client, then a compatibility matrix and release automation. These are network clients. In-process Go and TypeScript bindings are a separate design ([Embedded SDKs](https://tomiok.github.io/persiaDB/guide/sdks/)). The checkboxes are M16 (target v1.1), blocked on v1.0. ## M15 — Benchmarks, hardening, v1.0 [Section titled “M15 — Benchmarks, hardening, v1.0”](#m15--benchmarks-hardening-v10) Depends on M1 through M14 as a release, with room to benchmark after M7. A reproducible harness and a regression gate, SIMD and other work driven by profiles, soak and chaos tests, a security review, and the format 1.0 freeze: open questions closed as ADRs, golden fixtures for every released version, a documentation site (embedded, server, cloud, Kubernetes, relaxation tuning, operations), and the `v1.0.0` release. This guide is source material for that later site (ROADMAP 15.5.4). M0–M15 are the MVP. The live percentage in [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) counts only those leaves. Milestones after M15 are post-MVP and are not in the snapshot table above. ## After v1.0 [Section titled “After v1.0”](#after-v10) | Target | Milestones | Depends on | | ------ | --------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | v1.1 | **M16** Embedded SDKs, **M17** Highlighting and snippets, **M18** Autocomplete, **M19** Import, **M31** Persia Studio | M15, plus the feature’s own deps | | v1.2 | **M20** Encryption at rest, **M21** Python SDK | M20 on storage (M2, M12) and v1.0 (M15); M21 on the network clients (M14) and the C ABI (M16) | | v2.0 | **M22** Multi-tenant, **M23** Persia Cloud, **M24** CDC connectors, **M25** Web dashboard | M22 before M23; M23 also needs Kubernetes (M13) and encryption (M20) | | v2.1 | **M26** UI components | Server HTTP, highlighting, autocomplete | | v2.2 | **M27** AI search | Format 1.0, blobs, snippets | | v2.3 | **M28** On-device | The C ABI, and a storage backend without mmap | | v3.0 | **M29** Geo, **M30** Enterprise | M29 needs query (M7) and the format freeze (M15); M30 needs Kubernetes (M13), encryption (M20), tenancy (M22), and the dashboard (M25) | Multi-tenant is numbered before Persia Cloud because the hosted service is one file per tenant. v2.1, v2.2, and v2.3 are separate targets so a UI kit does not wait on embeddings, and embeddings do not wait on phones. Geo has no dependency on those three; it stays on v3.0 with the enterprise format work. The leaf list is [ROADMAP.md](https://github.com/tomiok/persiaDB/blob/main/ROADMAP.md). ## M16 — Embedded SDKs (v1.1) [Section titled “M16 — Embedded SDKs (v1.1)”](#m16--embedded-sdks-v11) Blocked on M15. Go via cgo and `@persiadb/node` via N-API, both through `persia-ffi`, as [ADR-0018](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0018-embedded-sdk-bindings.md) and [`docs/sdk/`](https://github.com/tomiok/persiaDB/blob/main/docs/sdk/README.md) already specify. This page does not repeat that design. The network clients stay M14. The browser build is M28. ## M17 — Highlighting and snippets (v1.1) [Section titled “M17 — Highlighting and snippets (v1.1)”](#m17--highlighting-and-snippets-v11) Blocked on M7 and M15. SPEC §7.7 already plans `explain` inside M7. Highlights are the post-v1 half: fragments sliced from the stored field, off unless the query asks. ## M18 — Autocomplete (v1.1) [Section titled “M18 — Autocomplete (v1.1)”](#m18--autocomplete-v11) Blocked on analysis (M4), the prefix walk (M8), the embedded API (M10), and v1.0. A `suggest` call returns completions from edge n-grams or from the term dictionary. It is not a search, and it does not run relaxation. ## M19 — Import (v1.1) [Section titled “M19 — Import (v1.1)”](#m19--import-v11) Blocked on the CLI (M10) and v1.0. `persia import` gains Elasticsearch, Meilisearch, and a SQLite FTS5 dump. JSONL import is already leaf 10.4.2. The SQLite path does not link `rusqlite`. ## M20 — Encryption at rest (v1.2) [Section titled “M20 — Encryption at rest (v1.2)”](#m20--encryption-at-rest-v12) Blocked on local and cloud storage and on v1.0. SPEC §1.2 leaves encryption to the disk for v1. This milestone encrypts bytes Persia writes, behind an incompatible format flag, including objects in the bucket. The key is not in the file. ## M21 — Python SDK (v1.2) [Section titled “M21 — Python SDK (v1.2)”](#m21--python-sdk-v12) Blocked on the C ABI (M16) and on the network-client conformance suite (M14). The embedded package calls `persia.h`. A smaller HTTP client covers programs that will not load a native library. Neither one reimplements the engine. ## M22 — Multi-tenant mode (v2.0) [Section titled “M22 — Multi-tenant mode (v2.0)”](#m22--multi-tenant-mode-v20) Blocked on cloud storage and v1.0. One `.persia` file per tenant, backups to S3, and a server route that cannot see another tenant’s file. Per-document principals are M30. ## M23 — Persia Cloud (v2.0) [Section titled “M23 — Persia Cloud (v2.0)”](#m23--persia-cloud-v20) Blocked on Kubernetes (M13), encryption (M20), and multi-tenant files (M22). A hosted control plane with a free tier. Local reads stay local. The cloud project is the source of truth after sync. The operator UI is the M25 dashboard, not a second app. ## M24 — CDC connectors (v2.0) [Section titled “M24 — CDC connectors (v2.0)”](#m24--cdc-connectors-v20) Blocked on the server and v1.0. A separate process tails Postgres, MySQL, or Supabase (Supabase is the Postgres connector) and writes through the public API. Nothing in the query path talks to the database. ## M25 — Web dashboard (v2.0) [Section titled “M25 — Web dashboard (v2.0)”](#m25--web-dashboard-v20) Blocked on the server and v1.0. An index explorer, a query playground, editors for boosts, synonyms, and pin/hide rules, and a list of queries that stayed empty. Auth is the server’s API keys. ## M26 — UI components (v2.1) [Section titled “M26 — UI components (v2.1)”](#m26--ui-components-v21) Blocked on the HTTP API, highlighting, and autocomplete. Search box, suggestions, hits, and facets for React, Vue, and Svelte, sharing one HTTP client. These packages do not embed the engine. ## M27 — AI search (v2.2) [Section titled “M27 — AI search (v2.2)”](#m27--ai-search-v22) Blocked on blobs, v1.0, and snippets. A vector field (SPEC §18.6), hybrid ranking, an embedding model shipped in the repo, a read-only MCP server, and retrieval of passages from one file. Writing the answer stays outside the process. ## M28 — On-device (v2.3) [Section titled “M28 — On-device (v2.3)”](#m28--on-device-v23) Blocked on the storage trait and the C ABI. iOS and Android link `libpersia`. WASM waits on a storage backend with no mmap and no POSIX fsync, which ADR-0018 already requires and which this milestone builds. ## M29 — Geo filters and sorting (v3.0) [Section titled “M29 — Geo filters and sorting (v3.0)”](#m29--geo-filters-and-sorting-v30) Blocked on query and v1.0. A `geo_point` field, a bounding box, a circle, and sort by distance. Nothing here depends on M26–M28. The filters are not relaxed. ## M30 — Enterprise (v3.0) [Section titled “M30 — Enterprise (v3.0)”](#m30--enterprise-v30) Blocked on Kubernetes (M13), encryption (M20), multi-tenant mode (M22), and the dashboard (M25). Per-document allow and deny rules, an audit log, OpenID Connect login, and an on-premises bundle that boots without calling Persia Cloud. ## M31 — Persia Studio (v1.1) [Section titled “M31 — Persia Studio (v1.1)”](#m31--persia-studio-v11) Blocked on the TypeScript SDK (M16). A read-only client that runs in the browser and does not talk to a server. The core is the WASM build that SDK work shares. The user picks the `database.persia/` directory with the File System Access API, which Chromium-based browsers provide. Firefox and Safari do not, and this version does not add a server for them. The page browses segments and the manifest, shows documents and tombstones, runs queries with the BM25 score breakdown, and shows WAL state. Write mode and a local `persia studio` server are out of scope. It is the seed of the v2.0 dashboard (M25). # Embedded SDKs > The planned in-process Go and TypeScript bindings. Not implemented. Network clients are a different milestone. **Planned. Not implemented.** There is no Go module and no npm package. The Rust API these bindings will wrap does not exist either: [`persia`](https://github.com/tomiok/persiaDB/blob/main/crates/persia/src/lib.rs#L1-L4) exports no types. Nothing on this page opens a database. This page is the short version. The design is [ADR-0018](https://github.com/tomiok/persiaDB/blob/main/docs/adr/0018-embedded-sdk-bindings.md) and [`docs/sdk/`](https://github.com/tomiok/persiaDB/blob/main/docs/sdk/README.md). When this page and the SPEC disagree, the SPEC wins. ## Two different SDKs [Section titled “Two different SDKs”](#two-different-sdks) SPEC §13 names both, and they are easy to mix up. | | Embedded bindings | Network clients | | -------------- | ------------------------------------------------------------------------------- | ---------------------------------------------------- | | Who calls them | A Go or TypeScript program that wants a file, the way Rust will call `Db::open` | A program talking to `persia-server` | | Paths | `sdk/persia`, `sdk/node` (neither directory exists) | `sdk/rust` (`persia-client`), `sdk/go` (`persia-go`) | | Milestone | M16 (target v1.1), after v1.0. Design: ADR-0018. Blocked on M15. | M14, after the server (M11) and sharding (M13) | | Status | Design only | Not started | The network clients are described on [Server, cloud, and Kubernetes](https://tomiok.github.io/persiaDB/guide/server/). ## Recommendation [Section titled “Recommendation”](#recommendation) One engine, the Rust one (ADR-0001). Go and TypeScript do not grow a second search implementation. * **Go** links a static library through cgo. The library is `persia-ffi`, a C ABI crate that does not exist yet. A pure-Go engine, and a WASM engine hosted by wazero, were rejected: durability is fsync and a writer lock (SPEC §4.5, §8.4), and mapped reads are already part of storage (ADR-0017). * **TypeScript** ships first as an N-API addon (`@persiadb/node`) for Node.js, Bun, and Deno. The addon calls the same Rust functions the C ABI calls. A WASM package for browsers and edge isolates comes later, only after a storage backend exists that does not need mmap or POSIX fsync. That backend is ROADMAP M28 (target v2.3). Prebuilt binaries are how a caller builds without Rust: a checksummed `libpersia.a` for Go, and per-platform optional npm dependencies for Node. Until format v1.0 there is no compatibility promise (SPEC §15). The first binding release follows the v1.0 tag (M15, then M16). ROADMAP 10.5.2’s `v0.1.0-alpha` tag is the core’s notice that the format is still unstable; it is not the binding cut. ## The call, once it exists [Section titled “The call, once it exists”](#the-call-once-it-exists) Illustrative. These are not functions in a repository. ```go db, err := persia.Open(ctx, "app.persia", persia.WithCreate()) products, err := db.CreateCollection(ctx, schema) err = products.Put(ctx, doc) err = products.Commit(ctx) results, err := products.Search(ctx, persia.Text("runing")) ``` ```ts const db = await open("app.persia", { create: true }); const products = await db.createCollection(schema); await products.put(doc); await products.commit(); const results = await products.search("runing"); ``` `Put` of an existing id is the update (SPEC §8.3). There is no `BEGIN` / `ROLLBACK`: multi-document transactions are out of scope (SPEC §1.2). A put is specified to be visible to a later search on that database before commit (SPEC §8.2). Commit is the durable manifest. A second writer fails immediately (SPEC §8.4). Search is BM25F (SPEC §7.2). A query that does not match is relaxed — synonyms, dropped terms, prefix, edit distance, then a fallback — and the response says what changed (SPEC §7.6). That cascade is M8. M10 depends on M8, so the first core release that the bindings can wrap already includes it. The bindings do not implement it themselves. Empty hits are still a successful response when relaxation is off or the budget runs out. Go methods take `context.Context` and return `error` values with `errors.Is` sentinels. TypeScript methods return promises and throw `PersiaError` with a `code`; `get` of a missing id returns `null`. Neither context nor `AbortSignal` can cancel a call that has already entered the engine. The engine is synchronous, and it has no cancel flag. The query budget is `relax.budget_ms` on the query. Pagination is a page per call (`limit` / `offset`, or a `search_after` cursor, SPEC §7.4). There is no iterator that borrows the index across the language boundary. Documents cross the boundary as UTF-8 JSON in the shapes SPEC §6.2 and §7.1 sketch. Go structs and TypeScript objects are encoded by the language package. The core stores CBOR (SPEC §6.4). A fault on a mapped page kills the process (ADR-0010). Mapped reads are opt-in and off by default (ADR-0017). A Rust panic is caught at the boundary and becomes an error; the handle is then poisoned. ## What has to exist first [Section titled “What has to exist first”](#what-has-to-exist-first) The bindings are blocked on the Rust API (ROADMAP 10.1): `Db::open`, collections, put, delete, get, commit, search, a stable error type. The first release a caller is supposed to use is v0.1 (M10). That milestone is blocked on M6 (in progress), M7 (in progress), M8 (not started), and M9 (not started). Blobs in the SDK wait on the same work, plus the upload-staging decision (SPEC §18.8). Opening `s3://` waits on cloud backends (M12). The WASM package waits on a storage backend nobody has designed. Live status is [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md). The phased table is in the [design index](https://github.com/tomiok/persiaDB/blob/main/docs/sdk/README.md). # Server, cloud, and Kubernetes > The planned Persia DB gRPC and HTTP server, S3, GCS, and Azure Blob storage, Kubernetes sharding, and the Rust and Go network SDKs. **Planned.** Nothing here is running. The server milestone (M11) starts after the embedded API. Cloud backends (M12) and distribution (M13) come after the server. SDKs (M14) follow the protocol. ## Server [Section titled “Server”](#server) gRPC is the primary API; HTTP/JSON mirrors it for tooling. Both are specified to track `persia.proto` (SPEC §12.1–§12.2). Generated Rust code is specified to be committed so a build does not need `protoc` (ADR-0003). That code is not in the tree: `persia-proto` is module docs only (M11). The service sketch in the SPEC is illustrative: ```proto service Persia { rpc CreateCollection(CreateCollectionRequest) returns (Collection); rpc Put(stream PutRequest) returns (PutResponse); // docs, optional blob chunks rpc Delete(DeleteRequest) returns (DeleteResponse); rpc Get(GetRequest) returns (Document); rpc Search(SearchRequest) returns (SearchResponse); // includes relaxation info rpc GetBlob(GetBlobRequest) returns (stream BlobChunk); // supports ranges rpc Commit(CommitRequest) returns (CommitResponse); rpc Stats(StatsRequest) returns (StatsResponse); } ``` Responses carry the manifest sequence they were served from, so a client can ask for read-your-writes with `min_seq`. Errors map to canonical gRPC codes. TLS is optional and on by configuration, with API keys or bearer tokens, and optional mutual TLS (SPEC §12.3). Embedded mode stays unauthenticated. Health checks are `/livez` and `/readyz` (ready means recovery has finished). Shutdown drains, flushes, and drops the writer lock or lease. Observability is `tracing` spans, OpenTelemetry export, and a Prometheus `/metrics` endpoint: query latency by relaxation level, cache hit ratio, segment counts, merge backlog, WAL bytes, replica lag, and blob throughput (SPEC §12.4). Logs are structured JSON, and secrets are not logged (SPEC §17). The container image is planned as distroless, non-root, with a read-only root filesystem, built in CI for amd64 and arm64 with an SBOM (ROADMAP 11.5, SPEC §11.6). ## Cloud [Section titled “Cloud”](#cloud) Opening `s3://…`, `gs://…`, or `az://…` selects the backend (SPEC §10.2). The remote layout under a prefix is immutable manifests, segments, tombstone sidecars, WAL chunks, blobs, and a `LEASE` object (SPEC §10.4). Commit is put-if-absent of the next manifest. Each provider’s conditional create is part of the work (MinIO `If-None-Match`, GCS generation match, Azure `If-None-Match`), and a backend that cannot do it fails clearly (ROADMAP 12.1.4). Provider quickstarts and a note on request cost are ROADMAP 12.6, written when the backends exist. Development emulators are wired in the integration environment at M2 and filled in at M12 (ROADMAP 2.5, 12.2.4). ## Kubernetes and sharding [Section titled “Kubernetes and sharding”](#kubernetes-and-sharding) A cluster is a set of shards. Each shard is its own database under `/shard-/`. Documents route by `hash(id) mod shard_count` using xxh3. The shard count is fixed when the collection set is created; splitting shards is a post-v1 design that takes advantage of immutable segments (SPEC §11.1, §18.5, ROADMAP 13.6). Roles, all in `persia-server` (SPEC §11.2): * **Writer** — holds one shard’s lease, accepts writes, compacts, uploads segments. * **Reader** — serves queries from committed manifests, holds no shard lease, scales horizontally. * **Router** — scatters a query, merges top-k and facets. It may run in the same process. The lease carries a fencing token. Every manifest embeds that token. A writer whose lease has lapsed stops writing, and readers reject a manifest whose token went backwards (SPEC §11.3). Readers poll for the next manifest (default 1 second, with jitter) and swap snapshots atomically. Lag is `persia_replica_lag_seconds` (SPEC §11.4). Cross-shard scoring uses each shard’s own statistics by default. `global_stats: true` makes the router collect statistics before scoring (SPEC §11.5). The Helm chart planned in SPEC §11.6: a StatefulSet for writers (one pod per shard, a volume for cache and WAL), a Deployment plus HPA for readers, a headless Service for discovery, pod disruption budgets, readiness and liveness probes, and a ServiceMonitor. Workload-identity examples for AWS, GCP, and Azure are part of that milestone (ROADMAP 13.5). ## SDKs [Section titled “SDKs”](#sdks) These are the **network** clients. Embedded Go and TypeScript, which open a file in-process, are a different design: [Embedded SDKs](https://tomiok.github.io/persiaDB/guide/sdks/) (ADR-0018). Neither family is implemented. Rust (`sdk/rust`, crate `persia-client`) and Go (`sdk/go`, module `persia-go`) sit on the same protocol (SPEC §13.2). Each adds connection pooling, retries with backoff on idempotent calls, deadlines, batching, blob streaming, and shard-aware writes. A shared, language-neutral conformance suite runs against both SDKs and the server (ROADMAP 14.1). A compatibility matrix is part of the SDK milestone (ROADMAP 14.4). Both clients are part of that milestone, after the protocol exists. # Roadmap > Milestone status for Persia DB, read from PROGRESS.md when this site is built. M0–M15 are the MVP (v1.0). Later milestones run through v3.0. Work is a tree of milestones. `M0` through `M15` are the MVP and ship as v1.0. Every leaf is one issue and one pull request. This table is read from [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) when the site is built: 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. That file is regenerated from checkboxes in [ROADMAP.md](https://github.com/tomiok/persiaDB/blob/main/ROADMAP.md) and is the live status. The ready-next list lives there too. A milestone is **done** when every leaf is done, **in progress** when some are done, **ready** when its dependencies are done and its own leaves have not started, and **blocked** while a dependency is unfinished. | Milestone | Delivers | Depends on | Snapshot | | ------------------------------------------- | ---------------------------------------- | ----------------------------------------------------------- | ------------------ | | **M0** Project bootstrap | Workspace, CI, tooling, docs scaffolding | — | In progress, 35/36 | | **M1** Format primitives | Encodings, codecs, header, frames | M0 | Done, 53/53 | | **M2** Storage abstraction & local backends | Storage, memory, local file, container | M1 | Done, 46/46 | | **M3** Schema, documents, collections | Collections, documents, manifests | M1 | Done, 15/15 | | **M4** Text analysis | Tokenizers, stemmers, analyzers | M0 | Done, 29/29 | | **M5** Segment writer & reader | Segment writer and reader | M1, M3, M4 | Done, 48/48 | | **M6** Write path, durability, merging | WAL, commit, merge, recovery | M2, M5 | In progress, 49/58 | | **M7** Query engine & ranking | Parser, BM25F, top-k, facets | M5 | In progress, 46/51 | | **M8** Zero-results avoidance | The zero-results cascade | M7 | Blocked, 0/29 | | **M9** Blob store | Pack file, remote blobs, GC | M1, M2 | Ready, 0/20 | | **M10** Embedded API | Db and Collection, CLI, v0.1 | M6, M7, M8, M9 | Blocked, 0/19 | | **M11** Server | gRPC, HTTP, image | M10 | Blocked, 0/23 | | **M12** Cloud storage backends | S3, GCS, Azure, cache, replicate | M2, M6, M9, M11 | Blocked, 0/29 | | **M13** Distribution & Kubernetes | Lease, shards, router, Helm | M11, M12 | Blocked, 0/24 | | **M14** SDKs | Rust and Go network clients | M11, M13 | Blocked, 0/19 | | **M15** Benchmarks, hardening, v1.0 | Benchmarks, hardening, format freeze | M1, M2, M3, M4, M5, M6, M7, M8, M9, M10, M11, M12, M13, M14 | Blocked, 0/22 | What each milestone contains, in prose, is the [milestones page](https://tomiok.github.io/persiaDB/guide/roadmap/). ## After v1.0 [Section titled “After v1.0”](#after-v10) The snapshot above is M0–M15 only. [PROGRESS.md](https://github.com/tomiok/persiaDB/blob/main/PROGRESS.md) keeps that percentage on those leaves and reports M16–M31 separately. None of the later milestones have been started. | Target | Milestones | Delivers | | ------ | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | v1.1 | M16–M19, M31 | Embedded Go and TypeScript SDKs, highlighting, autocomplete, import from Elasticsearch, Meilisearch, and SQLite FTS5, Persia Studio (read-only, in the browser) | | v1.2 | M20–M21 | Encryption at rest, Python SDK | | v2.0 | M22–M25 | One file per tenant, Persia Cloud, CDC from Postgres, MySQL, and Supabase, web dashboard | | v2.1 | M26 | React, Vue, and Svelte search components | | v2.2 | M27 | Hybrid text and vector search, in-repo embeddings, MCP, single-file retrieval | | v2.3 | M28 | WASM, iOS, and Android | | v3.0 | M29–M30 | Geo filters and sort, document access control, audit log, SSO, on-premises packaging | Multi-tenant (M22) comes before Persia Cloud (M23) because the service is one file per tenant, and Cloud also waits on encryption (M20). The longer notes are the [milestones page](https://tomiok.github.io/persiaDB/guide/roadmap/). [Milestones and dependencies](https://tomiok.github.io/persiaDB/guide/roadmap/)[ROADMAP.md](https://github.com/tomiok/persiaDB/blob/main/ROADMAP.md) # Waitlist > Leave an email to hear when Persia DB can be run. The browser encrypts it before it is stored. Persia DB is pre-alpha. There is no release to install. Leave an email if you want a note when a build is worth trying. A name, a company, and a use case are optional. The page encrypts the form in your browser with a public key. What leaves the browser is ciphertext. The project keeps that ciphertext in a git branch, one line per submission. The private key is not on this site and not in the repository, so the stored lines are not readable from the repo alone. The waitlist is not open yet. The public key or the relay URL is not configured. EmailName (optional)Company (optional)Use case (optional) Leave this empty Join the waitlist JavaScript is required. Encryption runs in the browser, so this form does not post the address anywhere without it.