File format and storage
This page describes how the files are organized. Offsets, magic numbers, and CRC coverage are SPEC §4 and SPEC §5.
Local container
Section titled “Local container”<name>.persia starts with a 12 KiB prefix: a 4 KiB header page (magic, format versions, compatibility
flags, a database UUID) and two 4 KiB superblock pages. The slots are a double buffer on separate
pages, so one bad sector cannot lose both. Open reads both, throws away a slot whose CRC fails, and
uses the higher commit sequence (SPEC §4.2–§4.3). From byte 12288 the file is
append-only frames. Each frame has a kind, a length, a payload padded to 8 bytes, and a CRC32C.
A torn or corrupt frame ends recovery; the file is logically truncated there (SPEC §4.4).
Integers are little-endian. Headers and frames use CRC32C. Internal content hashes and shard routing use xxh3 with seed 0. Blob ids are SHA-256 (ADR-0002), a content address that stays stable for the life of the blob.
The compatibility rules (SPEC §15) are part of the design, and the bytes may still change before v1.0. An unknown major version or an unknown incompatible flag is a hard reject. An unknown read-only-compatible flag opens the file read-only. Every format change is a SPEC update, a version or flag bump, an ADR, and a golden fixture (see Contributing).
The container header, the superblock slots, and frames (encode, decode, scan, and the streaming encoder) are implemented in persia-format (ROADMAP 1.6 and 1.7).
persia-format also has the encodings those structures use, plus the compression codecs
below.
Segments and blocks
Section titled “Segments and blocks”A segment is a self-contained byte string: sections, then a footer with statistics, a section table, and a CRC (SPEC §5.1). It is a frame payload locally and a standalone object remotely, and the bytes match. Sections hold the term dictionary, postings, frequencies, skip lists, optional positions, length norms, columnar values, stored documents, the id map, and blob references (SPEC §5.3). Deleted ids are not a section.
╔═ storage ════════════════════════════════════════════════════════╗ ║ local file object storage ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ header, 12 KiB ┃ same bytes ┃ manifests/seq ┃ ║░ ║ ┃ two superblocks ┃ ┃ segments/... ┃ ║░ ║ ┃ append-only frames ┃ ┃ blobs/... ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ sections, then sections ┃ ║░ ║ ┃ footer: stats, table, CRC ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ 16 KiB blocks · codec None or Zstd ║░ ╚══════════════════════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
Large sections are fixed-size blocks, 16 KiB uncompressed by default. Each block carries its own CRC, codec, and uncompressed length, so object storage can fetch one block with a range request (SPEC §5.2). Codecs:
| Codec byte | Name | Meaning |
|---|---|---|
| 0 | None |
The payload is the raw bytes. Its length equals uncompressed_len. |
| 1 | Zstd |
Exactly one zstd frame whose declared content size equals uncompressed_len. |
Any other codec byte is corruption. The reader refuses a declared length above the caller’s maximum
before it allocates (the decompression-bomb guard, ADR-0005). Writers may store None when zstd
does not shrink the block. The zstd level is a writer choice and is not stored.
compress / decompress for those two codecs is done (ROADMAP 1.4). The on-disk block
header, the block table, and random-access reads are implemented (ROADMAP 1.5, SPEC §5.2).
Integer codecs that sit inside blocks (delta-coded doc ids, bit-packing, postings blocks,
frame-of-reference for numbers) are implemented and tested (ROADMAP 1.2, ADR-0004). Segment section
writers in persia-engine use them. persia-engine implements collection schema, documents, and manifests (M3 done, 15/15) and a segment builder, writer, and reader (M5 done, 48/48). Its write path is in progress (M6, 49/58): a memtable, a WAL with chunks, group commit, and replay, flush, commit, snapshots, auto-commit, and open checks, deletes and upserts across segments with tombstone sidecars, and segment merging (a merge policy, a segment merger, a background scheduler, and deletion of obsolete objects). Its query path is in progress (M7, 46/51): a query AST, user-text compiler, filter DSL, and query limits, collection statistics (document count, document frequency, average field length), BM25F scoring with static rank and score bounds, matching and score top-k over one view, and a collection search with sorts, pages, facets, and stored fields (CollectionViews::search). Relaxation has not been started (M8). A collection snapshot can be searched as library code. There is no search call on the database handle. The leaf count is on the roadmap.
One storage interface, several backends
Section titled “One storage interface, several backends”The engine is specified to see every backend as the same trait: get, ranged get, put, put-if-absent, list,
delete (SPEC §10.1). The Storage trait and the memory and local-file backends are implemented (M2).
S3, GCS, and Azure Blob are not (M12). The write path already uses the trait. Query execution that would search through it is not written.
| Backend | How a database is addressed | Milestone |
|---|---|---|
| Local file | app.persia, or file:///… |
M2 |
| Memory | memory:// |
M2 |
| S3 (and MinIO) | s3://bucket/prefix |
M12 |
| GCS | gs://bucket/prefix |
M12 |
| Azure Blob | az://account/container/prefix |
M12 |
Cloud credentials use each provider’s normal chain (environment, config files, instance metadata). Persia-specific configuration is an override, not a requirement. An endpoint override exists for the development emulators (MinIO, fake-gcs-server, Azurite).
A remote commit uploads new segments, then creates manifests/<seq+1> with put-if-absent
(SPEC §10.4). Conflict means another writer won. Readers discover the newest manifest by listing,
with an optional CURRENT hint. Remote reads go through a block cache (memory, and optionally local
disk) keyed by object and block (SPEC §10.6).
replicate mode keeps the local file as the primary and uploads committed manifests in order,
without blocking local writes on the network (SPEC §10.5). Remote readers of that replica are
eventually consistent; lag is a metric.