What is an embedded search database?

Status: pre-alpha. Persia DB is pre-alpha. The public persia crate has no Db::open. persia-engine can put, delete, and commit a document. persia-engine runs a ranked search over a collection snapshot: BM25F, filters, sorts, facets, and pages. persia::Db::open and search elsewhere on this site are illustrative, from SPEC.md. Counts below are read from PROGRESS.md when the site is built: 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. That file is the live counter. When this page and the SPEC disagree, the SPEC wins.

An embedded search database is a search library that runs in the same process as the application. You link it in. Queries are function calls in that process. There is no search daemon to install, and no separate server process that has to be up before a query can run.

Some embedded libraries keep the index in one file, the way SQLite does. Others keep a directory of segment files. Persia DB’s design is the single-file kind. A local database is <name>.persia, plus an optional <name>.persia.blobs pack (SPEC §4, §9). The planned call is Db::open("app.persia"), with no config file for the library (SPEC §14). That call is not in the crate.

Embedded full-text search in Rust is that in-process shape, as a Rust crate. Tantivy is a released library of that kind, and its index is a directory. Persia DB is a pre-alpha project aimed at one file, an engine written in this repository, and the same segment bytes in object storage. The design line is “SQLite for search”: open a file, index documents, and get ranked results that, by default, never come back empty (SPEC §1). The repository has no persia::Db search API yet (M10); the engine library code can search. How that empty-page rule is designed, and which query pieces exist, is Typo-tolerant search.

A search server is a different way to run search. You start it as its own process, send documents and queries over the network, and operate that service. Meilisearch and Elasticsearch are servers. SQLite FTS5 and Tantivy are libraries. Where Persia DB sits next to SQLite FTS5, Tantivy, and Meilisearch is the comparison. That page is not a benchmark.

When an embedded library fits

The choice is about where the index lives and who operates it.

An embedded search database fits when search belongs in the same program as the data:

• The program is one process (a CLI, a desktop tool, a single service, a test) and can open a file or a directory.
• The same binary that writes documents runs the query.
• For the single-file kind, the database you copy is that file. In the Persia DB design it is <name>.persia, plus <name>.persia.blobs when blobs are used (SPEC §4, §9).
• You do not want a search cluster for that data.

A search server fits when many applications share one index, when callers should use HTTP without linking a library, or when a team already runs that service.

The Persia DB design is the library, with a server only as a later option. The planned gRPC and HTTP server embeds the same engine as the library (SPEC §3, §12). That server is milestone M11. persia-server is a placeholder binary that exits with “not implemented (M11)”. See Server, cloud, and Kubernetes.

For a search feature in production today, use an engine that already answers queries. The comparison says which established tool fits which job. Persia DB has no persia::Db search API yet (M10); the engine library code can search.

How the Persia DB design approaches it

The design contract is SPEC.md (draft v0.1). The table is that design. The right-hand column is what the repository contains.

Design choice	Specified in	In the repository
In-process library. No daemon and no config file for the embedded API. Options are builders, with defaults.	SPEC §14	Planned (M10). The persia crate exports no Db and no Collection.
One local file, <name>.persia, plus an optional <name>.persia.blobs pack for client bytes such as PDF or images.	SPEC §4, §9	The container format and the single-file container (open, append, commit, recover, compact) are implemented (M1, M2). Done (M5, 48/48). Segments are searched through CollectionViews::search. Container commit is done (M2, 46/46). The document write path is in progress (M6, 49/58): WAL, flush, commit, and snapshots, deletes and upserts, and segment merging. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle. The blob store has not been started (M9). BlobId and BlobRef exist as types in persia-format.
The engine is written in this repository: analysis, BM25F, filters, facets, and query execution.	SPEC §6, §7. ADR-0001 forbids depending on Tantivy, Lucene, or SQLite.	persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle.
A query with too few hits is relaxed step by step (synonyms, dropped terms, prefix, a fuzzy step, then a fallback). The response names the step. On by default.	SPEC §7.6	Not started (M8). Ranking and relaxation settings exist as schema config structs. The cascade does not run. See Typo-tolerant search.
Segments are immutable. The same segment bytes live in a local file or in S3, GCS, or Azure Blob. Query nodes are stateless.	SPEC §3, §5, §10, §11	A segment writer and reader exist (M5). Memory and local-file storage exist (M2). Object-storage backends and Kubernetes are planned (M12, M13).
A power loss leaves the previous commit or the new one.	SPEC §4.5, §8	The container commit is done (M2, 46/46) and is crash-tested. The engine write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Database::open reloads the committed manifest, checks it against the superblock, and replays the WAL. Open and recovery orchestration is done (single-writer enforcement, persia verify, and the crash-loop harness). A collection snapshot can be searched as library code. There is no search call on the database handle.
Optional gRPC and HTTP server. Embedded mode has no authentication.	SPEC §12, §14	Planned (M11).
Embedded Go and TypeScript bindings open the same file in-process. Rust and Go network clients talk to the server.	SPEC §13, ADR-0018	Design only. No Go module and no npm package. Network clients are milestone M14.

The first release a caller is supposed to use is v0.1 (M10). That milestone is blocked on M6 (in progress), M7 (in progress), M8 (not started), and M9 (not started). The on-disk format stays unstable until v1.0 (M15, SPEC §15). The crates are unpublished (publish = false, version 0.0.0). There is no crates.io release.

SPEC §16.2 lists latency and throughput figures and labels them as targets, not promises. This project does not publish search benchmarks. There is no Db::search. The query compiler, the BM25F scorer, and matching with score top-k over one view are library code (ROADMAP 7.1–7.4.4). So are filter evaluation, sorts, and facets, through CollectionViews::search over a collection snapshot (ROADMAP 7.4.5, 7.5). A search call on the database handle is not.

Out of scope for v1 (SPEC §1.2): text extraction, OCR, and transcription (clients supply text; blob bytes stay opaque); vector or semantic search; SQL, joins, multi-document transactions, and cross-collection queries; more than one writer on a shard; encryption at rest inside the file.

What the code can do today

PROGRESS.md is the live counter: 321 of 521 roadmap leaves done (62%) in the 2026-11-05 PROGRESS.md snapshot, with 0 marked in progress. Milestone states are on the roadmap page, generated from that file. persia-format implements the on-disk codecs (M1 done, 53/53). persia-storage implements the Storage trait, memory and local-file backends, and the single-file container (M2 done, 46/46). S3, GCS, and Azure have not been started (M12). persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. persia-analysis implements tokenizers and the built-in analyzers de, en, es, fr, it, keyword, pt, standard, and whitespace (M4 done, 29/29). It does not search. persia-blob exports no API (M9 not started; its dependencies are done). The public persia crate exports no Db and no Collection. The crates are unpublished (publish = false, version 0.0.0). Building one segment is not persia::Db::open.

A git dependency can use:

• persia-format, which encodes and decodes on-disk bytes: integer codecs, checksums, None and Zstd block compression, block framing, the container header, the superblock, and frames.
• persia-storage, which implements the Storage trait, a memory backend, a local-file backend, and the single-file container.
• persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle.

The persia command lists and removes temporary and quarantine files next to a container or a local store (persia container files, persia container sweep, persia store files, persia store sweep). It can also inspect and dump one segment file (persia inspect segment, persia dump segment, with --verify on inspect). It does not import documents or search. There is no persia::Db::open. The intended API, marked as illustrative, is the embedded API page. The file layout is File format and storage.

Read next

• Typo-tolerant search — query relaxation, BM25F, and filters: design versus what the engine implements.
• Comparison — Persia DB next to SQLite FTS5, Tantivy, and Meilisearch. Not a benchmark.
• FAQ — short answers on status, the file format, crash safety, SDKs, and the license.
• How it works — crate layers, the write path, and the query path.
• Roadmap — milestone status. The live counter is PROGRESS.md. The longer dependency notes are Milestones.
• Overview — the project narrative, including goals and non-goals.
• Crate architecture — how the Rust crates are layered.