SPEC-CONCURRENCY.md

spec: SPEC-CONCURRENCY version: 2 status: draft-normative audience: implementers of any lock-taking mkit-core/mkit-cli code path; reviewers of concurrency-sensitive changes

SPEC-CONCURRENCY — the total mkit lock order

Status: Normative for lock naming, location, and acquisition order. Draft because §3.1 documents an open coordination gap this document does not resolve. See SPEC-CONVENTIONS §2 for what draft/normative mean.

Scope: this document is the single owner of the total lock order across every lock any mkit process takes — repo-local locks defined in SPEC-WORKTREE §4.3, the ref-history locks defined in mkit-core::refs, and the file-transport's own CAS lock defined in SPEC-TRANSPORT. Any other SPEC-*.md document that mentions a lock cross-references this one rather than restating the order; this document does not restate those other documents' non-locking content.

1. Why this document exists

Nearly a dozen SPEC-*.md documents each independently described a slice of mkit's locking model, using inconsistent names for the same lock and never stating the total acquisition order a multi-lock caller must follow. That let deadlock-shaped bugs (two processes taking the same two locks in opposite order) hide in the gap between documents. This spec is the one place that total order lives; every other document points here.

2. Lock inventory

The recovery log (.mkit/recovery-log) has no dedicated lock — see §3.2.

In the schematic names above, <ref> MUST be the lowercase hexadecimal BLAKE3 digest of the full UTF-8 ref path relative to the common directory, such as refs/heads/main. <branch> MUST be the same digest for refs/heads/ followed by the branch name. This bounds each filename independently of ref length and keeps namespaces distinct. History and mutation locks retain separate prefixes. Version 2 deliberately replaces escaped ref names with these digest names; only the current lock naming contract is supported.

3. Cross-subsystem interactions

3.1 File-transport CAS vs. local worktree operations (detected, not coordinated)

<root>/.mkit/refs/.lock serializes all file-transport Any, Missing and Match writes against other file-transport instances pointed at the same root. Direct local ref mutations instead share the full-ref <common_dir>/refs-<ref>.lock, including unconditional writes and deletes. Each domain therefore prevents its own unconditional writer from bypassing a concurrent conditional write. The two domains use different lock files: the file transport does not acquire the local full-ref lock, worktree.lock, worktrees.lock, or refs-history-<branch>.lock.

The cross-domain gap remains: local mkit commit/checkout/gc against a directory simultaneously served by mkit serve or mkit-server is not coordinated with the transport. When the served root is the repository's common dir (.mkit), the transport's <root>/refs/... files are the local refs, so a local ref mutation can race a client CAS. When the served root is any other directory the two sets of refs are separate files. In either layout a GC sweep cannot delete a pushed object: pushes store packs under <root>/packs/, which gc neither sweeps nor roots (SPEC-GC "Concurrent writers and the grace window"). The supported file-transport deployment is a bare/shared remote that a worktree-owning process does not also mutate directly. Local worktree commands against a live mkit serve or mkit-server root remain unsupported.

MKIT-11/#655 turned this from silent into detected, without closing it: every live server process on the root — mkit serve <path> (stdin SSH-frame, its only mode) and any server embedding mkit-server's filesystem layout — holds a shared kernel lock (std::fs::File::lock_shared, never exclusive — SPEC-TRANSPORT documents multiple concurrent serve processes against one root, e.g. one per SSH forced-command connection, as a supported deployment, so serve instances must not exclude each other) on serve.lock (in the common dir) for its whole lifetime. mkit-server also holds server.lock (in the common dir) exclusively, so one mkit-server serves a root at a time; it shares serve.lock with any mkit serve processes on the same root. The one exclusive hold: at startup mkit serve tries serve.lock exclusively, without waiting, and only while it holds it (so no other server is up) sweeps the temp files crashed uploads left in packs/, then drops it and takes the shared lock; if the lock is busy it skips the sweep. worktree.lock and worktrees.lock acquisition (mkit-cli's acquire_worktree_lock and acquire_worktrees_registry_lock) each follow up with a non-blocking exclusive probe of that same serve.lock (mkit_core::repo_lock::probe_exclusive); when the probe finds it busy (i.e. at least one serve is alive), the command prints a warning to stderr and proceeds anyway — it does not refuse or block. The paths that take worktree.lock without the helper call the same probe directly (mkit-cli's warn_if_served) right after acquiring it: the per-branch unpack-and-publish phase of fetch/pull and the fast-forward phase of pull (remote_dispatch::fetch_objects_inner and pull_all_with), and status's opportunistic index refresh. So every command that takes worktree.lock or worktrees.lock emits the warning: the worktree-mutating commands, gc (which takes both), tag (create), attest, fetch/pull, clean, and the fast-forward of mkit git pull. Paths that take neither lock do not, including push, tag -d, and the import phase of mkit git import/fetch/pull and mkit git export, which hold only git-<remote>.lock (§4's per-command table).

Residual gaps this warning does not close:

• A push made directly against the file transport (mkit push mkit+file:///path, bypassing mkit serve entirely) is not detected — nothing takes serve.lock on that path.
• Detection is one-directional: a mkit serve that starts during a local command's already-in-flight critical section is not itself warned, since serve does not probe the worktree locks.

3.2 Recovery-log record/expire synchronization

ops::recovery::record and ops::recovery::expire are not internally synchronized (see that module's own concurrency note) and have no dedicated lock of their own. Correctness instead relies on lock superset containment:

• Every producer (commit --amend, reset, rebase, stash pop) calls record while already holding its own tree's worktree.lock — the same lock it took for the read-modify-write the recovery entry is about.
• mkit gc calls expire while holding worktrees.lock and every registered tree's worktree.lock (SPEC-WORKTREE §4.3's holder list) — a strict superset of any single producer's one lock.

Because gc's lock set always contains whichever single worktree.lock a producer would be holding, a producer's record append and gc's expire rewrite can never interleave — without a separate recovery-log lock being necessary. ops::recovery's module doc calls this "the repo lock" generically rather than naming a specific lock file, precisely because the guarantee comes from lock containment, not from a dedicated primitive.

3.3 First-parent history publication

The history-enabled CLI uses refs::update_ref_with_ancestry, taking the branch history lock and then its full-ref mutation lock before reading state. Initial ancestry construction, multi-commit fast-forward, generation selection, intent persistence, ref publication and descriptor publication all run within these guards. Snapshot readers take the same pair, and reject pending intent. Guarded internal ref primitives MUST NOT reacquire the held mutation lock.

Pending intent retains both its old and target ref as GC roots, even in a build without history-mmr; all raw mutation paths reject stepping over that intent. The exact durable state machine and generation semantics are SPEC-HISTORY-PROOF §4. Legacy event-journal APIs remain explicit low-level interfaces and do not establish first-parent ancestry.

4. Global lock order and per-command lock sets

A process that takes more than one lock from §2 MUST acquire them in this order:

worktrees.lock  ≺  per-tree worktree.lock(s)  ≺  refs-history-<branch>.lock(s)  ≺  refs-<ref>.lock(s)

All direct local mutations of a full ref path MUST hold its refs-<ref>.lock, including unconditional writes and deletes. An Any condition removes the value precondition, never the serialization requirement. The lock is nested inside history guards by history-aware writers; guarded internal primitives MUST NOT reacquire it. Multiple locks within a class MUST be acquired in canonical order before proceeding to the next class: main worktree first then registry IDs, branch names lexicographically, and full ref paths lexicographically. Current rename operations publish individual refs; this order does not promise an atomic multi-ref transaction.

The git bridge locks nest only with the ref class of this chain: git-<remote>.lock ≺ git-import-key.lock ≺ refs-<ref>.lock(s). A process MUST NOT acquire either bridge lock while it holds worktrees.lock, a worktree.lock, a refs-history-<branch>.lock, or a refs-<ref>.lock. mkit git pull releases git-<remote>.lock when its import phase returns, before its fast-forward acquires worktree.lock.

The file transport's <root>/.mkit/refs/.lock serializes every write condition within that transport. It is not composed with the local chain above; the served-root/local-worktree deployment restriction in §3.1 still applies.

serve.lock is a lifetime detection guard, excluded from this acquisition chain. A server holds it shared while handling requests, including requests that acquire the file transport's refs/.lock. A local command only probes it non-blocking from inside its worktree.lock/worktrees.lock critical section and immediately releases a successful probe. Neither path waits for exclusive ownership of serve.lock, so detection introduces no blocking lock-order dependency. The per-command sets below omit these implicit probes (see §3.1).

Per-command lock sets (extends SPEC-WORKTREE §4.3's holder list with the history lock):

A process that violates this order and blocks on two locks acquired in opposite order by two racing processes will each time out independently (repo_lock's default 5s timeout) rather than deadlock indefinitely — but SHOULD NOT rely on the timeout as a substitute for correct ordering; a timed-out multi-lock command leaves its already-acquired locks released (RAII drop) but may leave mid-sequence on-disk state requiring the caller to retry.