Storage format

The storage format is the contract that makes content addressing work: the same logical object must always serialize to the same bytes, so that its address reproduces across machines and across TOVIO releases. This document defines how TOVIO objects are encoded to bytes, how those bytes are addressed, and how a repository is laid out on disk.

This page summarizes format version 1.

Invariants

Four guarantees hold for every conforming implementation:

Invariant	What it means
Determinism	The same logical object serializes to the same bytes on every platform, so its content address reproduces everywhere.
Self-description	Every structured object names its own format version and type in its bytes.
Verifiability	Every object is re-hashed and checked on read and on receipt; a mismatch is a corruption event, never silent.
Crash safety	A crash mid-write never corrupts the repository.

Encoding: canonical CBOR, raw payloads

TOVIO uses two encodings, and the choice is never ambiguous:

• Structured objects use deterministic CBOR (RFC 8949 §4.2): shortest-form integers and lengths, definite lengths only, map keys sorted bytewise with no duplicates, a single top-level value, and no trailing bytes. There is exactly one valid byte sequence for any given object — that is what makes the address reproducible.
• Opaque payloads — blob, chunk, and ciphertext bodies — are raw bytes, never CBOR-wrapped.

Strings are UTF-8. Binary fields (hashes, signatures, nonces, wrapped keys) are CBOR byte strings.

The object envelope

Every structured object is a CBOR map whose first two keys are the envelope header (shown here in field order, not encoded-byte order):

{
  "v": 1,        // format version (uint)
  "t": <tag>,    // object type tag (uint)
  ... type-specific fields ...
}

Both single-byte keys sort ahead of every longer key under bytewise ordering, so the header is always at the front — though on disk the order is "t" then "v", because the encoded key "t" (0x6174) sorts before "v" (0x6176). blob and chunk have no envelope — they are raw payload bytes. Their type is known from where they live in the store and from the object that references them, not from inline framing.

Addressing: BLAKE3-256

An object's address is the BLAKE3-256 hash of its stored bytes — the canonical CBOR for a structured object (envelope included), or the raw payload for a blob/chunk.

• A large file's identity is the address of its chunk-manifest, not a hash of the reassembled file.
• Addresses are compared as raw 32-byte values. The textual blake3:<64-hex> form is a UI and log convenience only — it never appears on disk or on the wire.

Chunking: FastCDC binary parity

Large files are split into content-defined chunks with FastCDC, so TOVIO handles binary files natively without an LFS bolt-on. Chunk boundaries are computed on content, so an edit to one region re-chunks only the affected neighborhood and unchanged chunks deduplicate. The chunking parameters (min/avg/max) are part of the repository's effective format identity for newly minted objects: fixed format-1 defaults apply unless the policy manifest carries an authenticated, commit-linked chunking override — never the replica-local config file — and every chunk-manifest records the parameters it was cut with, so historical files stay self-describing.

On-disk layout

A repository lives under .tovio/. Abridged to the load-bearing entries:

.tovio/
├── format            # "tovio-format 1" — the format-version marker
├── config.toml       # replica-local operational config (mode; [ref_tombstones], [gc],
│                     #   [line_ops]) — parsed fail-closed, never synced
├── identity/         # local identity keys (encrypted at rest; never synced)
├── objects/          # content-addressed store, sharded by address prefix and
│   │                 #   segregated by type (blob/ chunk/ chunk-manifest/ tree/ commit/
│   │                 #   policy-blob/ policy-chunk/ conflict/ behavioral/ audit-entry/
│   │                 #   obliteration/ policy-manifest/ capability/ symbol-shard/
│   │                 #   symbol-xref/ sealed-symbol-shard/ rationale/ …)
│   └── pack/         # optional packing layer — identity-transparent, no address changes
├── refs/             # CRDT mutable state: heads/ tags/ remotes/ + authenticated frontiers
├── op-log/           # local operation log (NOT synced) — backs `tovio undo`
├── policies/         # working copy of policy manifests (the object is authoritative)
├── keys/             # cached wrapped/derived keys (encrypted at rest; NOT synced)
├── forge/            # Forge-only control plane (NOT snapshotted)
├── working-copy/     # HEAD (current lane) + the current change id
└── semantic/         # optional derived symbol-graph cache + behavioral registry

Two boundaries are load-bearing for security and sync:

• objects/ is segregated by type, so a relay can enumerate, serve, and garbage-collect by class — and so policy (encrypted) objects are obviously distinct from clear ones. The type directory is authoritative for blob/chunk, which carry no inline tag.
• op-log/, keys/, forge/, and the checkout scratch state never enter a tracked snapshot. Secrets never enter objects/, so they never sync. (See the crypto envelope for key handling.)

Sharding splits each address into a one-byte prefix directory plus the remaining 31 bytes, capping directory fan-out. The optional compression and packing layer folds those loose files into objects/pack/ behind the same logical addressing, so no address ever changes. Symbol-graph objects (symbol-shard, symbol-xref) are synced content under objects/; the semantic/ directory holds only a derived cache, never a second source of truth.

Object lifecycle

• Write — serialize, hash the logical bytes, optionally codec-encode, write to a temp file in the same directory, fsync, then atomic-rename into place. The rename is the commit point; readers never see a partial object. Writing an object that already exists is a no-op (dedup).
• Read — locate the object (loose shard, then pack indexes), decode any codec, re-hash the logical bytes, compare to the requested address, then decode. A mismatch surfaces a corruption error with the address and path.
• Obliterate — replace a payload with a typed obliteration tombstone at the same logical address; reads thereafter return a typed "obliterated" response, never a 404.
• Garbage collection — tovio gc removes objects unreachable from any ref, recent op-log entry, or audit chain, and can compact by rewriting packs. It is conservative, never removes obliteration tombstones, and in a complete repository retains every reachable object.

Versioning

The .tovio/format marker and each object's "v" carry the format version. A backward-compatible addition (a new optional field or object tag) does not bump the version. Any change that alters the bytes — and therefore the address — of existing objects is a breaking version bump that follows the public deprecation policy with a documented migration. A chunking override is validated (min < avg < max, within the documented ranges) before any object is written, never rewrites existing objects, and may temporarily reduce deduplication — the authoring surface warns about that.

Conformance in one line

For every fixture in the shared corpus, the encoder must produce byte-identical canonical encodings, compute identical addresses, round-trip decode→encode with no byte changes, and reject malformed or non-canonical inputs. See the conformance page for the corpus itself.

Last reviewed September 9, 2026

Suggest an improvement to this page  ·  Not for security reports — see disclosure