Crate architecture
The picture below is the intended layering in SPEC §3.
persia-format, persia-storage, persia-engine, and persia-analysis contain library code.
persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle.
persia exports no Db. persia-server is a placeholder binary that exits with “not implemented (M11)”. persia-blob and persia-proto are placeholders, module docs only.
docs/architecture.md is the contributor map (crate graph, write path, query
path) and stays the place for that detail.
╔═ layers ═════════════════════════════════════════════════════════╗ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-server, Rust SDK, Go SDK ┃ ║░ ║ ┃ planned ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia · Db and Collection ┃ ║░ ║ ┃ stub · no methods yet ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ engine ┃ ┃ blob ┃ ┃ analysis ┃ ║░ ║ ┃ WAL and query ┃ ┃ chunks, dedup ┃ ┃ analyzers ┃ ║░ ║ ┃ in progress ┃ ┃ not started ┃ ┃ done ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-storage ┃ ║░ ║ ┃ memory and local file ┃ ║░ ║ ┃ s3, gcs, azure planned ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ persia-format ┃ ║░ ║ ┃ encodings, checksums, codecs ┃ ║░ ║ ┃ done ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ╚══════════════════════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
Three rules shape the rest of the design:
- Immutable segments and append-only metadata. Crash recovery, caching, and replication all read a snapshot instead of locking a mutable index.
- The same logical objects everywhere. A segment, manifest, WAL chunk, or blob is a named object. Locally they are frames in one file; remotely they are one object each. The segment bytes do not change between the two.
- The engine is synchronous. It talks to I/O only through the
Storagetrait. Async belongs in the cloud backends and in the server.
| Crate | Role | Status |
|---|---|---|
persia-format |
Encodings, checksums, frames, block codecs. No I/O policy. | Implemented (M1) |
persia-storage |
Storage trait, memory, local file, later S3 / GCS / Azure. |
Memory, local file, and the container are implemented (M2). Cloud is M12. |
persia-analysis |
Tokenizers, normalizers, stemmers, analyzers. | Done (M4). The query compiler runs them. This crate does not search. |
persia-engine |
Schema, segments, WAL, commit, merge, query, BM25F, relaxation. | persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. |
persia-blob |
Content-addressed chunks, dedup, garbage collection. | No API yet (M9) |
persia |
The embedded API: Db and Collection. |
Not implemented (M10). The crate exports no types. |
persia-proto |
persia.proto and committed generated code. |
Module docs only (M11) |
persia-server |
gRPC and HTTP. Roles: writer, reader, router. | Placeholder: exits with “not implemented” (M11, M13) |
persia-cli |
Sweep temporary and quarantine files. inspect segment and dump segment read one segment (--verify checks it). compact, bench, import, and search are later. |
Sweep, inspect, dump, and verify exist. The rest is M10. |
Dependencies point downward only (persia-format at the bottom). cargo xtask check-deps enforces
the graph. The public facade orders blob writes before the document commit so the engine never has
to know about blob bytes (SPEC §9.5).
On a query, the design is: analyze the text, search each segment and the memtable, merge the per-segment top-k, and if there are too few hits run the relaxation cascade. The response includes which level matched. The compiler, the BM25F scorer, matching, and the top-k merge exist as library code. The cascade does not. See Typo-tolerant search.