Concepts

Extensibility

How PTWM accepts new codecs, transforms, classifiers, and other contributions as signed third-party bundles.

PTWM ships a fixed set of built-in codecs and transforms — Huffman, rANS, Zstd, Identity, Order-1 ScaleAC, PerGroupCodebook, and 16 preprocessing ops. They cover the production cases that the chain explorer fits onto modern checkpoints. Anything beyond that — a learned codec for a new dtype, an ablation variant, a domain-specific classifier, a custom delta scheme — plugs in through the extension system.

Why an extension system

Two pressures drove the design:

Research velocity. Iterating on a new codec shouldn't mean forking PTWM core. The dispatcher lets a research codec compete in the same trial-encode loop as the built-ins.
Ablation honesty. Removing a codec for an ablation study should remove it cleanly — same wire format, same dispatcher, same measurement harness. The policy file consumed by ptwm bench compress --variant constrains which canonical IDs are eligible per run; see trust and security for the schema and resolution semantics.

The 13 contribution kinds

PTWM groups extensible surfaces into a sealed taxonomy of 13 kinds. Each kind is one wire-format hook plus one Rust trait / Python Protocol.

Kind	What it does
`transform`	Reversible byte-level preprocessor (bit reorder, byte split, …).
`plane_codec`	Per-plane entropy coder (Huffman, rANS, learned codec, …).
`chain_builder`	Emits candidate `Chain`s for a `(dtype, role)` pair.
`chain_explorer`	Generative search over chains beyond what builders cover.
`classifier`	Decides a tensor's `ClassifierRole` (weight, scale, …).
`scorer`	Content-aware quality signal feeding the dispatcher.
`delta_scheme`	Residual encoder for reference-frame compression.
`integration_adapter`	Drop-in glue for a third-party loader (transformers, …).
`container_layout`	Alternative on-disk layout for the multi-tensor blob.
`hardware_backend`	GPU / accelerator decode path.
`benchmark_metric`	Custom column for `ptwm bench compress` results.
`training_hook`	Compress-aware callbacks during fine-tuning.
`raw_binary`	Schema-tagged passthrough for non-tensor blobs.

The kind set is sealed — adding a 14th would be a wire-format change. The members are unsealed within each kind: anyone can ship a new plane_codec without touching PTWM.

Three flavors

Every contribution is one of:

WASM (wasm32-wasip1). Sandboxed by Wasmtime with fuel + memory limits and zero host imports by default. The portable, security-first choice. Performance ceiling ~30% below native cdylib for typical byte-level work.
Native cdylib (.so / .dylib / .dll). Loaded via libloading with a signature check first. No sandbox. Use for performance-critical or hardware-backed paths.
Python host. Pure-Python contributions registered via the ptwm.extensions entry-point group. Useful for classifiers and integration adapters that don't have a hot loop.

Each kind declares which flavors it accepts in docs/architecture/extension-system.md.

Behavioral equivalence

A WASM, native, and host implementation of the same contribution must produce byte-identical output for the same input. PTWM enforces this with a per-kind round-trip test in the conformance harness. The guarantee lets the dispatcher swap flavors transparently — useful for moving from a portable WASM build to a faster native build without re-signing the bundle.

Identity, trust, lifecycle

Every contribution has a 32-byte CanonicalId derived from blake3(author_pubkey ‖ 0x00 ‖ name ‖ 0x00 ‖ version). Built-ins use a sentinel pubkey (BUILTIN_PUBKEY = [0xBB; 32]) and are trusted by construction. Third-party contributions are signed (Ed25519) and verified against the user's keyring on every container open; see the trust and security guide.

Lifecycle modes control state reuse across calls:

none — every call is independent; init is implicit.
thread — one state per rayon worker thread, reused for the worker's lifetime.
process — one state for the whole process. The contribution is responsible for thread-safety.