ARCHITECTURE.md

In docs/ · At 619cbad

Preview

Text · 12.6 KiB

ARCHITECTURE — mkit code map

Status: Informative. Audience: contributors and integrators who need to find where something lives and how the pieces connect.

This document is the index. Normative shape lives in SPEC-*.md.

1. Workspace layout

The Rust workspace lives under rust/. Crate boundaries match responsibility boundaries — there is no "common" or "utils" crate.

Crate	Path	Purpose
`mkit-core`	`rust/crates/mkit-core/`	Object store, packs, refs, index, worktree, ignore, repo lock, ops, signing, protocol framing, `verify`/`verify::closure` (partial/full disclosure)
`mkit-attest`	`rust/crates/mkit-attest/`	JCS canonical JSON, in-toto v1 Statement, DSSE envelope, signers, verifiers
`mkit-keystore`	`rust/crates/mkit-keystore/`	Key vault interface and backends (`specs/SPEC-KEYSTORE.md`)
`mkit-git-bridge`	`rust/crates/mkit-git-bridge/`	Git import/export bridge incl. fork mode (`specs/SPEC-GIT-BRIDGE.md`)
`mkit-rpc`	`rust/crates/mkit-rpc/`	Shared stdio framing for subprocess protocols (`specs/SPEC-RPC.md`)
`mkit-transport-memory`	`rust/crates/mkit-transport-memory/`	In-process transport, used by tests
`mkit-transport-file`	`rust/crates/mkit-transport-file/`	Local-filesystem transport, atomic CAS via `link(2)` on POSIX
`mkit-transport-http`	`rust/crates/mkit-transport-http/`	(legacy, superseded by `mkit-transport-connect`; see SPEC-TRANSPORT §5) reqwest plus rustls transport with bearer auth and `If-Match` CAS
`mkit-transport-connect`	`rust/crates/mkit-transport-connect/`	ConnectRPC client implementing `mkit.transport.v1.TransportService` (`specs/SPEC-TRANSPORT-CONNECT.md`); the transport `mkit-cli` actually constructs for `mkit+https://` / `mkit+http://`
`mkit-transport-s3`	`rust/crates/mkit-transport-s3/`	Hand-rolled SigV4 transport (R2 plus S3-compatible)
`mkit-transport-ssh`	`rust/crates/mkit-transport-ssh/`	Spawns system `ssh(1)`; framed protocol over stdio
`mkit-transport-enc`	`rust/crates/mkit-transport-enc/`	`mkit+enc://` no-OpenSSH encrypted transport, deprecated: no maintained server (`specs/SPEC-TRANSPORT-ENC.md`)
`mkit-sync`	`rust/crates/mkit-sync/`	Native clone/fetch/pull for `.mkit` repositories as a library: typed reports, no printing, no ambient config; `mkit clone`/`fetch`/`pull` are thin wrappers over it
`mkit-cli`	`rust/crates/mkit-cli/`	The `mkit` binary; thin glue over the library crates
`mkit-wasm`	`rust/crates/mkit-wasm/`	WASM bindings for browsers and Cloudflare Workers
`mkit-fuzz`	`rust/fuzz/`	cargo-fuzz harnesses (`docs/FUZZ.md`)
`mkit-sign-file` (contrib)	`contrib/signers/mkit-sign-file/`	Reference external signer; protocol conformance baseline
`mkit-sign-se` (contrib)	`contrib/signers/mkit-sign-se/`	Apple Secure Enclave signer (Swift / CryptoKit)
`mkit-sign-tpm` (contrib)	`contrib/signers/mkit-sign-tpm/`	TPM 2.0 P-256 signer (Linux/Windows)
`mkit-sign-ctap` (contrib)	`contrib/signers/mkit-sign-ctap/`	FIDO2/WebAuthn roaming-authenticator signer

The contrib/signers/* crates form a separate sibling Cargo workspace under contrib/signers/ — they are not members of the rust/ workspace (out-of-tree members break release-plz publishing). They share package metadata, lint config, and dependency pins by inheriting from contrib/signers/Cargo.toml, and they are not part of the mkit binary's link graph. They communicate over the protocol defined in specs/SPEC-EXTERNAL-SIGNER.md.

2. Data flow: object → pack → ref → transport

write path                                read path
==========                                =========

bytes                                     transport
  │                                         │
  ▼                                         ▼
chunker (FastCDC)                         pack reader
  │   SPEC-FASTCDC                          │   SPEC-PACKFILE
  ▼                                         ▼
object encoder                            object decoder + verify
  │   SPEC-OBJECTS §2 prologue              │   BLAKE3 verify on read
  ▼                                         ▼
content-addressed store                   refs
  │   .mkit/objects/<dd>/<hex62>            │   SPEC-REFS
  ▼                                         ▼
pack writer                               worktree materialize
  │   SPEC-PACKFILE                          │   symlink containment
  ▼                                         ▼
transport                                 user
  └─── SPEC-TRANSPORT 7-verb wire ──────────┘

Every stage validates the previous stage's output. Decoders refuse oversized allocations; the FastCDC chunker is fully deterministic (seed MKITFCDC); object readers re-hash bytes before returning.

3. Merkle-root object identity

Blob, Commit, Remix, and Tag take their object ID from a flat BLAKE3 digest of their canonical bytes. Tree and ChunkedBlob do not: their ID is the domain-bound root of a Binary Merkle Tree (BMT) built over their parts — one leaf per entry for a Tree, and a metadata leaf plus one leaf per chunk for a ChunkedBlob. The construction is normative in docs/specs/SPEC-MERKLE-OBJECTS.md; the vendored BMT lives in rust/crates/mkit-core/src/merkle.rs.

A flat digest names exact bytes and nothing more. Deriving the ID from a Merkle root over the object's parts adds two properties. Inclusion is provable: a caller holding one chunk of a ChunkedBlob, or one entry of a Tree, can prove it belongs to the object without fetching the rest. Completeness is structural: the store verifies these two types on read by decoding, recomputing the root, and comparing it to the ID, so "every child is present, unmodified, and in order" falls out of the check that already runs on each read — no trusted manifest involved. The root is also wrapped in a per-type domain, so identical leaf streams — an empty Tree and an empty ChunkedBlob, for example — never share an ID.

The guarantee composes up the object graph. A Tree's entry leaves bind the IDs of its children, a Commit binds its root tree, and every ref resolves to a commit. Verifying a commit against its ID therefore verifies every tree, entry, and chunk reachable from it.

ObjectStore::open enforces addressing consistency through the .mkit/format repository marker. ObjectStore::init writes the marker with the value bmt-v1; open requires that same value and rejects the repository with IncompatibleRepoFormat when the marker is missing or different, rather than misreading objects under the wrong addressing rule. The marker governs object identity only: the serialized wire format and the schema_version field are independent of it.

Verification. The BMT inclusion proofs this section's identity scheme makes possible are exposed as a first-class capability, not just an internal read-time check: mkit_core::merkle's id-based proof verifiers plus mkit_core::verify/verify::closure (partial and full disclosure against a commit id, for an external party with no repository access — a DA provider, a light client, a browser) round out the picture the diagram above starts. See docs/VERIFY.md for the user-facing guide and docs/specs/SPEC-MERKLE-OBJECTS.md / docs/specs/SPEC-DISCLOSURE.md for the normative wire formats and verification algorithms.

4. Attestation flow

Attestations are a separate resource class from objects. A commit without attestations and a commit with five are indistinguishable at the commit layer.

predicate JSON  ── caller-supplied
       │
       ▼
in-toto v1 Statement   (subject = commit BLAKE3, predicateType = URI)
       │   SPEC-ATTESTATIONS §4.2
       ▼
JCS canonical bytes    (RFC 8785; mkit-attest hand-rolled writer)
       │
       ▼
PAE("application/vnd.in-toto+json", payload)
       │   SPEC-ATTESTATIONS §2.1
       ▼
Signer ────► sig bytes        Signer is one of the two implemented paths:
       │                        - repo-key  (Ed25519, .mkit/keys/default.key)
       │                        - external  (subprocess; SPEC-EXTERNAL-SIGNER)
       │
       │                      Roadmap, not implemented (`SigstoreSigner`
       │                      returns `Error::SigstoreNotImplemented`):
       │                        - sigstore-keyless (Fulcio/Rekor)
       ▼
DSSE envelope ─► JCS ─► BLAKE3 → attestation id
       │
       ▼
.mkit/attestations/<commit-hex>/<att-id>.dsse
       │
       ▼
transport (OP_UPLOAD_ATTESTATION / DOWNLOAD / LIST)

mkit verify-attest walks the envelope, dispatches each signature to a verifier keyed on keyid, and consults trust roots from $XDG_CONFIG_HOME/mkit/trust-roots.toml (see THREAT-MODEL.md §5).

5. External signer protocol

The contract is docs/specs/SPEC-EXTERNAL-SIGNER.md (Protocol v1, v1.1 adds optional WebAuthn wrapping). Shape:

mkit attest ──► spawn binary at attest.external_signer_path
            ──► stdin:  one-line JSON {"pae_base64": …, "algorithm": …}
            ──► stdout: one-line JSON {"keyid": …, "sig_base64": …}
            ──► exit 0 on success

Argv comes from attest.external_signer_args (user-scoped config — see THREAT-MODEL.md §4) or per-invocation flags. Every reference signer in contrib/signers/ is a conformance test for this contract.

6. Where to start reading

Pick the closest match.

"I'm modifying the parser for an on-disk object."

Start at rust/crates/mkit-core/src/serialize.rs, then read docs/specs/SPEC-OBJECTS.md. Update the matching golden vector under rust/tests/golden/. If you change the wire shape, version it.

"I'm adding a new transport."

Start at rust/crates/mkit-core/src/protocol.rs for the Transport trait, then read docs/specs/SPEC-TRANSPORT.md for the verb set. Use mkit-transport-memory as the smallest implementation reference.

"I'm adding a new signer algorithm or implementation."

For an in-tree algorithm, edit mkit-attest's Algorithm enum and the verifier dispatch. For an out-of-tree signer (HSM, KMS, hardware token), implement docs/specs/SPEC-EXTERNAL-SIGNER.md and ship a binary — see contrib/signers/mkit-sign-file/ for a minimal example.

"I'm changing the wire format."

Bump the spec version in the relevant docs/specs/SPEC-*.md, write a new golden vector alongside the old one, and gate the new shape behind a version field. Do not delete the old vector until a major release window.

"I'm changing key handling, signing, or trust roots."

Read docs/THREAT-MODEL.md first. Crypto and key-handling changes require a second reviewer per CONTRIBUTING.md, plus a CHANGELOG entry under ### Security and a note on which threat-model section the change affects.

7. Cross-references

specs/SPEC-OBJECTS.md — on-disk object format
specs/SPEC-MERKLE-OBJECTS.md — Tree/ChunkedBlob BMT-root addressing
specs/SPEC-PACKFILE.md — packfile wire format
specs/SPEC-DELTA.md — delta encoding
specs/SPEC-FASTCDC.md — content-defined chunking
specs/SPEC-REFS.md — ref names, CAS variants
specs/SPEC-INDEX.md — repo-local index
specs/SPEC-TRANSPORT.md — seven-verb wire protocol
specs/SPEC-SIGNING.md — commit / remix signing
specs/SPEC-ATTESTATIONS.md — DSSE plus in-toto v1
specs/SPEC-EXTERNAL-SIGNER.md — external signer subprocess protocol
SSH-SECURITY.md — SSH trust model (informative)
THREAT-MODEL.md — security boundaries (informative)
FUZZ.md — fuzz harness conventions

Preview

ARCHITECTURE — mkit code map

1. Workspace layout

2. Data flow: object → pack → ref → transport

3. Merkle-root object identity

4. Attestation flow

5. External signer protocol

6. Where to start reading

"I'm modifying the parser for an on-disk object."

"I'm adding a new transport."

"I'm adding a new signer algorithm or implementation."

"I'm changing the wire format."

"I'm changing key handling, signing, or trust roots."

7. Cross-references

Files

ARCHITECTURE — mkit code map

1. Workspace layout

2. Data flow: object → pack → ref → transport

3. Merkle-root object identity

4. Attestation flow

5. External signer protocol

6. Where to start reading

"I'm modifying the parser for an on-disk object."

"I'm adding a new transport."

"I'm adding a new signer algorithm or implementation."

"I'm changing the wire format."

"I'm changing key handling, signing, or trust roots."

7. Cross-references