SAG-Structured Capability v0.1

Status: v0.1 design baseline. This document defines the OCTX sag-structured/0.1 Capability and its complete data constraints.

New to SAG? Read the SAG introduction first to understand the differences among the SAG application, the retrieval architecture, and zleap-sag.

1. Purpose

sag-structured means that the Package explicitly carries complete, verifiable document → chunk → event → entity data that can be mapped directly to the SAG structural layer. It is one holistic Capability: it does not split chunks, events, and entities into three declarations, and it is not a statement that a local index is already ready.

Declaration:

{
  "capabilities": {
    "sag-structured": {"version": "0.1"}
  }
}

Vectors remain optional for this Capability. When vectors are carried, vectors/0.1 is declared separately.

2. Contents

OCTX + sag-structured/0.1
  -> chunks
  -> events
  -> entities
  -> chunk-event relations
  -> event-entity relations

vectors/0.1 -> depends on sag-structured/0.1

• All five structure files must appear and validate as one unit.
• Chunks, events, entities, and relations are no longer declared as separate Capabilities.
• sag-structured must be declared in the manifest and cannot be inferred automatically from the presence of files.
• vectors/0.1 is an independent optional Capability, but it may reference only valid sag-structured/0.1 records.

3. Chunks v0.1

File: data/chunks.jsonl

Each line is one JSON object:

{
  "id": "019c2222-2222-7222-8222-222222222222",
  "document_id": "019c1234-5678-7abc-8def-0123456789ab",
  "ordinal": 0,
  "text": "The first chunk"
}

Within one document_id, every ordinal must be unique. Gaps are allowed, and consumers restore order by ascending numeric value. Physical JSONL line numbers do not express Chunk order.

v0.1 uses document_id + ordinal as the standard source locator and does not simultaneously require page numbers, character offsets, or heading paths. Additional locators may be preserved as unknown optional fields, but cannot replace required fields.

4. Events v0.1

Files:

• data/events.jsonl
• relations/chunk-events.jsonl

Event:

{
  "id": "019c4444-4444-7444-8444-444444444444",
  "title": "Passing Five Passes and Slaying Six Generals",
  "content": "Guan Yu passed a series of checkpoints while escorting his two sisters-in-law to find his brother."
}

A top-level Event omits both parent_id and level and is treated as level 0. A child Event must provide both, with level = parent.level + 1. Parent-child relations must not form cycles.

An Event does not embed chunk_id, references, or a local rank. All provenance is expressed exclusively by chunk-events.jsonl.

Chunk-Event relation:

{
  "chunk_id": "019c2222-2222-7222-8222-222222222222",
  "event_id": "019c4444-4444-7444-8444-444444444444"
}

• Both referenced records must exist.
• (chunk_id, event_id) is the relation identity and must not be duplicated within the file.
• Every Event must be associated with at least one Chunk.
• One Chunk may be associated with one or more Events.
• When publishing Events without fine-grained chunking, the producer must create a real full-text Chunk that covers the entire document.

5. Entities v0.1

Files:

• data/entities.jsonl
• relations/event-entities.jsonl

Entity:

{
  "id": "019c5555-5555-7555-8555-555555555555",
  "name": "Guan Yu",
  "type": "person"
}

normalized_name, database entity_type_id, and local index fields are not part of OCTX. A consumer must preserve the producer's original type. It may map that type locally, but cannot reject a Package because the type is unknown.

Event-Entity relation:

{
  "event_id": "019c4444-4444-7444-8444-444444444444",
  "entity_id": "019c5555-5555-7555-8555-555555555555",
  "weight": 1.0,
  "description": "The main participant in the event"
}

(event_id, entity_id) is the relation identity and must not be duplicated within the file. Every Entity must be referenced by at least one Event.

6. Complete SAG-Structured Coverage

sag-structured/0.1 requires the complete chain to contain no orphan records:

1. Every Concept Document has at least one Chunk.
2. Every Chunk appears in at least one Chunk-Event relation.
3. Every Event appears in at least one Chunk-Event relation.
4. Every Event appears in at least one Event-Entity relation.
5. Every Entity appears in at least one Event-Entity relation.

If any requirement is not met, the entire sag-structured/0.1 Capability is invalid. The OCTX format itself may remain valid, but there is no separately valid chunks, events, or entities sub-capability.

This Capability does not accept the following synthetic fallbacks:

• Temporarily treating a Document as a Chunk.
• Temporarily treating a Chunk as an Event.
• Generating a synthetic record that exists only to satisfy the schema.
• Copying text from the preceding layer to fill a missing layer.

A missing layer must be produced by a real chunking or extraction process.

7. Vectors v0.1

Configuration: vectors/config.json

{
  "model": "example/embedding-model",
  "revision": "2026-07-01",
  "dimensions": 1024,
  "normalized": true,
  "fingerprint": "sha256:ef7eae8ff494be507243324a6aff67ef25ae8c4b400de6a3ca960280b0cf9aba"
}

• model is a required non-empty string.
• revision is an optional non-empty string.
• dimensions is a required positive integer. The vector dimensions of every target Arrow file must match it.
• normalized is a required Boolean that declares whether the packaged vectors were normalized when generated.
• fingerprint is a required sha256: digest identifying the complete embedding space and generation pipeline.
• A Package contains at most one configuration shared by all packaged Arrow files.
• API addresses, keys, service endpoints, and retrieval distance metrics are not stored.

To calculate fingerprint, copy the complete configuration, remove the fingerprint field, encode it with RFC 8785 JCS, and compute SHA-256. A producer must include every setting that affects vector output or query compatibility in the configuration, such as an immutable model revision, input prompt or template, pooling, truncation rules, output dimensions, and normalization. These additional fields also participate in the fingerprint. A consumer may reuse packaged vectors only when its local embedding pipeline computes exactly the same fingerprint; comparing only the model name is insufficient.

Standard target files:

• vectors/chunks.arrow
• vectors/events.arrow
• vectors/entities.arrow

A Package may include only some targets, but every target file that appears must completely cover all records in the corresponding JSONL file. Partial vector sets are not allowed.

An Arrow IPC file contains at least:

record_id: utf8 non-null
vector: fixed_size_list<float32>[dimension] non-null

Requirements:

• record_id is unique and covers the target record IDs exactly.
• No nonexistent record may be referenced.
• dimension is a positive integer and is consistent within a file.
• Vector elements must not be null, NaN, positive infinity, or negative infinity.
• A consumer may ignore additional columns, but they cannot replace required columns.
• vectors v0.1 does not use Arrow IPC body compression, avoiding uncontrolled decompression allocation before resource limits take effect.

Vectors participate in per-file digests and the Package Digest. Regenerating only the vectors still creates a new Release; when knowledge and structure have not changed, existing record IDs are preserved.

When the local embedding pipeline's fingerprint differs from the Package, the consumer discards the packaged vectors and rebuilds the corresponding targets completely. It must not truncate, pad, or mix them. model, revision, dimensions, and normalized support interpretation and diagnosis but cannot replace the fingerprint decision.

8. Validation Failures and Rebuilding

The entire sag-structured/0.1 Capability is invalid when any record in the standard JSONL files has one of the following problems:

• A required field is missing.
• A field type or value does not conform to the schema.
• An ID is duplicated.
• A reference does not exist.
• A relation combination is duplicated.
• A hierarchy contains a cycle or has an inconsistent level.

An importer must not silently skip bad records.

Handling rules:

• Valid OCTX format without a declared sag-structured: install normally; a complete structure may be generated locally from Markdown.
• Valid OCTX format with a declared but invalid sag-structured: report an error; importing the original structural layer is prohibited, and the declaration cannot be removed silently. Only after an explicit user choice may the complete structure be discarded and rebuilt locally.
• Invalid vectors/0.1: preserve valid knowledge and SAG structure, and rebuild only the corresponding vectors.
• Invalid OCTX format: installing the original Asset is prohibited.

Rebuilding does not patch individual records. If any SAG structure layer is invalid, rebuild the complete structure. If vectors are invalid, rebuild only the corresponding target:

Locally rebuilt results belong to the Installation and do not modify or retroactively prove the original Package valid. Redistributing enhanced results requires a derived Asset and a new Package.

9. Capability and ready

sag-structured is a static, verifiable Package Capability; ready is a runtime state of a particular consumer's local Installation.

• A valid sag-structured Package may still require local indexing because its vectors are incompatible.
• A Package containing only OCTX documents may reach ready after local construction, but the original Package does not thereby become sag-structured.
• Agent retrieval can be enabled only after the indexes required by the current Installation are actually ready.

10. Export

By default, zleap-sag export_octx() writes the structural layer only when every complete-coverage constraint is satisfied, and declares sag-structured/0.1 in capabilities.

• A partial structural layer must not enter the Package.
• A user may explicitly set documents_only=True to export only Markdown.
• Vectors are optional, but every target actually written must be complete.
• When locally enhanced content is exported again after import, it must create a new Asset with asset.derived_from.