Guides
Random access
Open a `.ptwm` bundle and fetch individual tensors without decompressing the whole file.
The .ptwm container ships with a per-tensor index. Opening a bundle and
fetching one tensor touches only that tensor's bytes — useful for partial
loading, sharded inference, or extracting a single weight from a large
checkpoint.
Open and fetch by name
from ptwm.random_access import TensorIndex
idx = TensorIndex.open("model.safetensors.ptwm")
weight = idx.get_tensor("layer_5.weight")
idx.names() returns the list of available tensor names. The index is
small (a manifest of name → offset records), so opening a 30 GB
.ptwm and fetching one tensor reads only a few hundred KB from disk.
Stream multiple tensors
for name, tensor in idx.stream_tensors(["embed.weight", "lm_head.weight"]):
...
stream_tensors issues one decode per requested tensor in name order. It
does not hold the whole bundle in memory.
When random access matters
| Use case | Why it helps |
|---|---|
| Loading the embedding layer for tokeniser tests | Fetch one tensor; skip GB of irrelevant weights. |
| Inspecting weight statistics post-hoc | Open the bundle read-only, sample tensors, close. |
| Streaming weights to remote workers | Producer reads index once, sends per-tensor bytes on demand. |
For full-model loads, use the safetensors integration
or patch_transformers() instead — they handle
the standard load_state_dict path.