Skip to content
Persia DB

How it works

Persia is a library you open on a file, plus a server that embeds the same engine. That library shape is an embedded search database: full-text search in the application process, stored in a file. The engine is synchronous. It talks to disks and object storage only through the Storage trait. Async belongs in the cloud backends and in the server. Memory and local-file backends exist. S3, GCS, and Azure do not.

╔═ layers ═════════════════════════════════════════════════════════╗ 
║       ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓       ║░
║       ┃ persia-server, Rust SDK, Go SDK                  ┃       ║░
║       ┃ planned                                          ┃       ║░
║       ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛       ║░
║                                ▼                                 ║░
║       ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓       ║░
║       ┃ persia · Db and Collection                       ┃       ║░
║       ┃ stub · no methods yet                            ┃       ║░
║       ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛       ║░
║                                ▼                                 ║░
║ ┏━━━━━━━━━━━━━━━━━━┓  ┏━━━━━━━━━━━━━━━━━━┓  ┏━━━━━━━━━━━━━━━━━━┓ ║░
║ ┃ engine           ┃  ┃ blob             ┃  ┃ analysis         ┃ ║░
║ ┃ WAL and query    ┃  ┃ chunks, dedup    ┃  ┃ analyzers        ┃ ║░
║ ┃ in progress      ┃  ┃ not started      ┃  ┃ done             ┃ ║░
║ ┗━━━━━━━━━━━━━━━━━━┛  ┗━━━━━━━━━━━━━━━━━━┛  ┗━━━━━━━━━━━━━━━━━━┛ ║░
║                                ▼                                 ║░
║       ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓       ║░
║       ┃ persia-storage                                   ┃       ║░
║       ┃ memory and local file                            ┃       ║░
║       ┃ s3, gcs, azure planned                           ┃       ║░
║       ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛       ║░
║                                ▼                                 ║░
║       ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓       ║░
║       ┃ persia-format                                    ┃       ║░
║       ┃ encodings, checksums, codecs                     ┃       ║░
║       ┃ done                                             ┃       ║░
║       ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛       ║░
╚══════════════════════════════════════════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
Layers from SPEC §3. Status from the 2026-11-05 PROGRESS.md snapshot. Format, local storage, schema, and text analysis are done (M1–M4). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. The blob store is not started (M9; its dependencies are done). The embedded API has not been started (M10), and the persia crate exports no Db or Collection. The server is not started (M11). Network SDKs are not started (M14). S3, GCS, and Azure are not started (M12). Arrows point down: each layer is used by the one above it.

A write

Section titled “A write”
╔═ write ══════════════════════════════════════════════════════════╗ 
║               ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓               ║░
║               ┃ 1. put or delete                 ┃               ║░
║               ┃ WAL, then memtable               ┃               ║░
║               ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛               ║░
║                                ▼                                 ║░
║               ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓               ║░
║               ┃ 2. flush                         ┃               ║░
║               ┃ memtable becomes a segment       ┃               ║░
║               ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛               ║░
║                                ▼                                 ║░
║               ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓               ║░
║               ┃ 3. commit                        ┃               ║░
║               ┃ data, then manifest, then        ┃               ║░
║               ┃ the inactive superblock          ┃               ║░
║               ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛               ║░
║                                ▼                                 ║░
║               ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓               ║░
║               ┃ 4. merge, later                  ┃               ║░
║               ┃ new segment, new manifest        ┃               ║░
║               ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛               ║░
╚══════════════════════════════════════════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
Local write path, partly in the tree (SPEC §4.5, §8.2). A crash before the superblock write keeps the previous commit. Deletions live in tombstone sidecars. The container commit is done (M2, 46/46) and is crash-tested. The engine write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Database::open reloads the committed manifest, checks it against the superblock, and replays the WAL. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle.

A query

Section titled “A query”
╔═ query ══════════════════════════════════════════════════════════╗ 
║               ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓               ║░
║               ┃ 1. analyze the text              ┃               ║░
║               ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛               ║░
║                                ▼                                 ║░
║               ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓               ║░
║               ┃ 2. each segment                  ┃               ║░
║               ┃ and the memtable                 ┃               ║░
║               ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛               ║░
║                                ▼                                 ║░
║               ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓               ║░
║               ┃ 3. merge the top hits            ┃               ║░
║               ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛               ║░
║                                ▼                                 ║░
║               ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓               ║░
║               ┃ 4. too few hits?                 ┃               ║░
║               ┃ no: return them                  ┃               ║░
║               ┃ yes: relax                       ┃               ║░
║               ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛               ║░
║                                ▼                                 ║░
║               ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓               ║░
║               ┃ 5. cascade, L0 to L5             ┃               ║░
║               ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛               ║░
╚══════════════════════════════════════════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
Designed query path (SPEC §7). The text compiler in step 1 and the BM25F scorer are in the tree. Searching segments, merging hits, and the cascade are not. Step 4 is the branch into zero-results relaxation. Filters stay strict unless the caller opts in. The response names the level and what changed. The longer account is typo-tolerant search. Text analysis is done (M4, 29/29). The query milestone is in progress (M7, 46/51). Zero-results relaxation is not started (M8).

The compiler and the BM25F scorer are in the tree. Searching segments and the relaxation cascade are not. Typo-tolerant search is the longer account.

Storage and segments

Section titled “Storage and segments”

A segment is one byte string: sections, then a footer. Locally it is a frame payload. Remotely it is one object. The bytes match. Codecs on a block are None and Zstd. The local container, the block codecs, and the memory and file backends exist. persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. S3, GCS, and Azure backends do not (M12).

╔═ storage ════════════════════════════════════════════════════════╗ 
║         local file                         object storage        ║░
║  ┏━━━━━━━━━━━━━━━━━━━━━━┓             ┏━━━━━━━━━━━━━━━━━━━━━━┓   ║░
║  ┃ header, 12 KiB       ┃  same bytes ┃ manifests/seq        ┃   ║░
║  ┃ two superblocks      ┃             ┃ segments/...         ┃   ║░
║  ┃ append-only frames   ┃             ┃ blobs/...            ┃   ║░
║  ┗━━━━━━━━━━━━━━━━━━━━━━┛             ┗━━━━━━━━━━━━━━━━━━━━━━┛   ║░
║                                ▼                                 ║░
║           ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓           ║░
║           ┃ sections, then sections                  ┃           ║░
║           ┃ footer: stats, table, CRC                ┃           ║░
║           ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛           ║░
║                16 KiB blocks · codec None or Zstd                ║░
╚══════════════════════════════════════════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
The same segment bytes in a local frame and in object storage (SPEC §5, §10). Deleted ids are not a section. Block framing is done (M1, 53/53). Memory and local-file backends are done (M2, 46/46). S3, GCS, and Azure are not started (M12).

The contributor map, with the crate dependency graph, is docs/architecture.md. Quotes from the code that exists today are on the code page.

Crate architectureRead the codeFile format