SPEC-HTTP-OBJECTS.md

spec: SPEC-HTTP-OBJECTS version: 1 status: draft-normative audience: implementers of HTTP object servers, caches, browsers, and disclosure verifiers

SPEC-HTTP-OBJECTS — HTTP object serving and proofs

Status: Draft, normative. The contract is pinned by the golden vectors; HTTP adapters and the span verifier follow in WP-4.12–4.16. Scope: repository selection, URL parsing, published-content resolution, HTTP representations, read authorization and admission, URL-token carriage, and browser access. This document changes no protobuf schema.

Authority: this document specifies the plain HTTP contract for issue #1088, WP-4.11. SPEC-CONVENTIONS supplies normative keywords. SPEC-TRANSPORT-CONNECT, abbreviated STC, owns repository identities and payment headers. SPEC-SERVER owns indexed mode, the published view, hooks, and durable outcomes.

1. Purpose and rationale

An object URL identifies immutable content. A ref-path URL follows a published ref and must revalidate. Both authorize within the selected repository before returning content, including on cache validation requests. A content digest alone grants no access to another repository's content.

User decision M4-a (2026-09-27) deliberately reinterprets PRD §6.6's statement that a Range request can include a byte-range disclosure proof: a proof range is selected by ?proof=1&range=a-b, never by Range. The proof is a separate representation with its own validator and media type. This lets a verifier request the complete proof without HTTP slicing it. User decision M4-b uses the MKDS container of existing MKDP v2 bundles (SPEC-DISCLOSURE, multi-chunk span section). MKDP payload kind 3 remains reserved; MKDP v2 bytes are unchanged.

2. Routes and URL syntax

Indexed mode is REQUIRED. An opaque deployment MUST NOT mount these routes; its normal missing-route response applies. The rules below apply to mounted HTTP serving routes. A single-repository deployment MAY omit the repository prefix. An explicit prefix MUST obey STC §7.4, including the bare-name restriction. HTTP selects the repository only by this path and MUST ignore X-Repository, Host, and forwarded headers as repository selectors. Adapters that validate routes before serving MUST use the pipeline's deployment namespace mode (SPEC-SERVER §6.2.2). Authority mode accepts opaque namespace names and rejects self-certifying owner IDs and root at both gates.

http-path    = repo-prefix "/-/" ( objects-form / refs-form )
repo-prefix  = "/" repository / ""   ; empty only in single-repository mode
repository   = <STC section 7.4 grammar>
objects-form = "objects/" object-id
object-id    = 64HEXLC
HEXLC        = DIGIT / %x61-66
refs-form    = ref-name "/-/" [ file-path ]
ref-name     = "refs/" ref-seg *( "/" ref-seg )
ref-seg      = <SPEC-REFS section 3 segment, excluding the delimiter "-">
file-path    = path-seg *( "/" path-seg )
path-seg     = 1*( unreserved / pct-encoded )
unreserved   = ALPHA / DIGIT / "-" / "." / "_" / "~"
pct-encoded  = "%" HEXDIG HEXDIG
pct-path     = [ file-path ]
url-token    = *<query-value octet other than "&" or "#">
query        = param *( "&" param )
param        = "proof=1" / "range=" 1*DIGIT "-" 1*DIGIT
             / "commit=" 64HEXLC / "path=" pct-path / "token=" url-token

Route and parameter literals above MUST match case-sensitively. Object and commit ids MUST use lowercase hexadecimal. A query is absent or nonempty; a trailing ?, empty parameter, unknown parameter, or repeated parameter MUST produce 400. Parameter names MUST NOT be percent-encoded. A token value is opaque at this stage: its token-format validation belongs to §3 step 5, so even an empty or malformed token MUST NOT cause a token-derived 400. Token-bearing vectors are deferred to WP-2.11/4.15.

A ref MUST satisfy SPEC-REFS §3, occupy at most 512 bytes, and begin refs/. Percent-encoding in a ref MUST produce 400. The ref ends at the first segment exactly -; the second /-/ is mandatory, even for the root tree. A valid stored ref with a - segment is not addressable by this grammar. For example, /-/refs/heads/-/topic/-/x addresses ref refs/heads and path topic/-/x, never refs/heads/-/topic. A - in a file path is an ordinary entry name.

Each file-path segment MUST be percent-decoded exactly once to one entry name under SPEC-OBJECTS §4.1. Invalid escapes, decoded separators, NUL, invalid names, empty segments, and leading or trailing path slashes MUST produce 400. Implementations MUST preserve decoded bytes, including non-UTF-8 bytes, with no Unicode normalization. %25, %3F, %23, and %20 represent literal percent, question mark, hash, and interior space. + MUST NOT stand for space. The joined decoded path, including separating / bytes, MUST be 1–1024 bytes; the empty path selects the root tree. A path= proof context uses the same encoding and permits an empty root path.

range=a-b selects inclusive content offsets. Both integers MUST fit u64, and a <= b; failures MUST produce 400. range requires proof=1. On an object URL, proof=1 requires both commit and path, including path= for the root. commit and path supply that object's proof context; on a ref URL they do not override the ref's resolution. Without proof=1 they do not change the object representation. Bounds against content size, selector applicability, and proof caps are checked later as 416, not 400. Every 400 MUST depend only on URL text and the deployment's route grammar, never on stored state. An HTTP Range parse failure MUST NOT produce 400.

The mandatory /-/ segment, which RPC service paths never contain, distinguishes this namespace from /.well-known/, /mkit.transport.v1.*, and /grpc.*. Routers MUST dispatch HTTP object routes by that segment, or RPCs by exact service path, never by a /grpc.* prefix glob. The key document in §6 is a separate well-known route. Routers MUST retain the original escaped path until this parser runs; framework path decoding MUST NOT reinterpret delimiters.

3. Response precedence

The server MUST apply these steps in order, taking the first terminal result. HEAD follows the same checks as GET but MUST send no body on every status.

Unsupported proof profile. A deployment without HTTP proof serving MUST validate the full query grammar at step 3 and apply repository existence, private-token validity, Authorizer, ordinary published target resolution and takedown checks at steps 5–8. If those checks permit the read, any proof=1 query MUST then receive the same 416 with Cache-Control: no-store, before proof-context validation, canonical representation metadata, selector sizing, validators, Admission, payment reservations or response-stream setup. HEAD MUST send no body. No proof capability may be advertised by this profile. commit, path and query range are checked only for syntax; their stored context, applicability and bounds do not affect this refusal. Ordinary published target resolution and takedown checks still produce their normal 404/451/503 results. In particular, If-None-Match: * MUST NOT turn this unsupported 416 into 304. Step 7's proof-context checks and steps 9–12 for successful proof representations apply only in a proof-capable profile. The shipped core and Workers HTTP handlers use the unsupported profile.

All errors MUST carry no-store in Cache-Control; in a bearer-gated deployment they MUST carry private, no-store. The missing-repository and missing/invalid-private-token 404s MUST be byte-identical in status, headers, and body for otherwise equivalent requests (including CORS and HEAD). Their response MUST NOT identify which check failed. The missing-repository and private-repository 404 paths SHOULD take uniform time. Hooks and storage failures MUST fail closed under SPEC-SERVER; an infrastructure failure is 503, never a fabricated success or content-dependent 400.

4. Published resolution and reachability

The caller's view for these requests MUST always be the published view (SPEC-SERVER, published view contract). Signed-read HTTP GETs are deferred. Pending content MUST NOT be served over HTTP, even to a writer or a token holder. Ref-path resolution MUST peel tags to a commit or remix and walk its tree by exact decoded entry bytes. Symlinks MUST be served as blob content; they MUST NOT be followed as filesystem paths.

A ref path is reachable by construction. For an id URL, the server MUST check repository membership first, with a cheap uniform 404 on a miss unless this repository has a tombstone for that exact id. A tombstone candidate MUST then prove reachability from a published ref value before step 8 can return 451; a failed proof is the same 404 as any other miss. A proof that cannot finish within the walk or decode caps is a failed proof for every non-writer reader: the same 404, and the same absent answer from the server-side object reader, as an unknown id. Only an authorized writer view may receive a typed exhaustion error. Ordinary member ids also prove reachability. The walk follows repository-local object references, including parents and chunk manifests, but MUST NOT descend through a blocked or tombstoned manifest; it MUST NOT follow foreign remix sources or pack-only delta bases. An id blocked globally but not yet tombstoned in this repository MUST return 404 at step 7, even when a cached reachability result or matching If-None-Match would otherwise produce 304. The walk MUST use current tombstones even when its reachability result was cached. A global blocklist entry alone is not permission to answer 451: the id MUST be reachable in this repository's published tree walk and have this repository's tombstone (SPEC-SERVER §14.5). The global content store MUST NOT be consulted to decide membership, reachability, or existence; it MAY supply bytes for an id already resolved in this repository and held by it. Extracted copies keyed by object id (SPEC-SERVER §9.6) MUST NOT constitute authorization, membership, reachability, or observable delta bases.

A rewind or ref deletion MUST invalidate reachability within a configured reachability_lag bound. Tombstones MUST take effect immediately, including on cached walks. Informative: external CDN copies of orphaned public objects persist until their max-age expires; purges occur on takedown, suspension, and visibility change, not every rewind or deletion.

In a proof-capable profile, for an object proof URL, commit MUST be reachable in the published view, and its decoded path MUST resolve to the requested object id. Either failure is 404. A ref proof uses its resolved commit. Informative: an MKDP/MKDS proof authenticates the path through trees to the commit and its signature; the reader-side ref-to-commit binding remains the server's claim. SPEC-SERVER §15 storage receipts attest only live committed advances to writers. A signed reader-side binding is a later follow-up.

5. Representations, validators, and headers

5.1 Object content and ordinary ranges

Blob responses MUST carry raw content; ChunkedBlob responses MUST carry concatenated chunk content in manifest order, not the manifest. Both use Content-Type: application/octet-stream on object-id routes. Successful ordinary ref-path file responses (200/206, including HEAD) MUST instead select Content-Type from the last decoded path segment's final extension, compared case-insensitively against this fixed allowlist. Content bytes MUST NOT be sniffed. This applies equally to public, private and URL-token requests.

SVG, HTML, HTM, XHTML, XML, JS, MJS and CSS MUST NOT receive their real media types. Each such ref-path file response MUST carry Content-Disposition with both filename="<ASCII fallback>" and filename*=UTF-8''<encoded last decoded segment>. The extended filename MUST preserve the decoded segment's octets, including non-UTF-8 octets permitted by §2, without normalization or a second URL decode. Every octet outside RFC 5987 attr-char MUST be percent-encoded. The ASCII fallback MUST retain only [A-Za-z0-9._-], replacing every other octet with _, and MUST be capped at 255 bytes. Header values are derived and fully percent-encoded outside attr-char, with a separately sanitized fallback; no raw request text is interpolated.

These rules MUST NOT change object-id routes, proof representations, non-file objects, errors, 402 or 304 responses. The 304 header list in §5.3 is unchanged: Content-Type and Content-Disposition need not be repeated. Validators, bytes, ranges, caching, admission and token bindings are unchanged.

Other object types MUST carry canonical object bytes with Content-Type: application/vnd.mkit.object. Pack-only Delta objects MUST NOT be served (SPEC-OBJECTS §1).

200 and 206 MUST carry ETag: "<64hex leaf id>", Accept-Ranges: bytes, X-Mkit-Object: <64hex leaf id>, and X-Mkit-Object-Type using the SPEC-OBJECTS §1 lowercase type name. Ref-path responses MUST also carry X-Mkit-Commit: <64hex resolved commit id>. Content-Length MUST describe the selected body, including for HEAD (the GET body length).

An ordinary single byte range MUST produce 206 with Content-Range: bytes a-b/N. Open-ended and suffix byte ranges follow HTTP byte-range semantics. A multi-range request MUST be served as 200 with the full representation; multipart responses are not supported. An invalid or unsupported Range header MUST be ignored. The server MUST honor If-Range: only a matching strong ETag enables slicing; a weak, nonmatching, or unsupported date validator yields the full 200 response. If-None-Match uses HTTP weak comparison, including *, against the selected representation's ETag, after authorization and resolution and before Range.

5.2 Proof selection

This section defines the proof-capable profile. The shipped HTTP handlers use §3's unsupported profile; they validate query syntax and refuse before any of the preparation or successful representation behavior below. The generic MKDP/MKDS formats and client verification remain supported.

?proof=1 MUST return an MKDP Object bundle, including canonical manifest bytes for a ChunkedBlob and canonical tree bytes for an empty root path. ?proof=1&range=a-b MUST return an MKDP Range bundle when the inclusive range lies within one chunk or a plain blob; otherwise it MUST return MKDS. The server MUST prove absolute offsets for chunked MKDP ranges with the complete preceding chunk-length proof set. Query ranges MUST NOT be clipped: any endpoint outside content is 416. The inclusive query becomes offset=a, len=b-a+1 using checked arithmetic; overflow is 416. Non-blob leaves do not support proof ranges and yield 416.

MKDP uses application/vnd.mkit.disclosure; MKDS uses application/vnd.mkit.disclosure-span. Proofs MUST carry Accept-Ranges: none, MUST ignore every HTTP Range/If-Range header, and MUST use 200 rather than 206. The 64 MiB encoded bundle cap applies. A deployment SHOULD configure a lower requested-content cap, for example 8 MiB. The requested-content range-length cap MUST be checked before the encoded-size cap; exceeding either yields 416 before Admission. A proof's declared_bytes MAY be computed exactly from indexed metadata without constructing the proof. The encoded-size cap MUST likewise be established from indexed metadata before Admission. The server MUST build the proof only after Admission allows it. Informative: complete preceding chunk-length proofs make ranges beyond roughly 30,000 preceding chunks (about 2 GiB of file) exceed the 64 MiB bundle cap and return 416. WP-4.14a's boundary-aware builder reduces reads and peak memory by retaining only the needed chunk bytes. It does not reduce the encoded proof size: the complete preceding proof set grows with the first chunk index. An oversized proof still returns 416. Changing that proof-size limit would require a separate format change.

The proof ETag MUST be "<commit>.<leaf>.<selector>", with lowercase 64hex ids and selector object or range-a-b (minimal decimal inclusive endpoints). It is distinct from the ordinary object ETag. The object and commit metadata headers from §5.1 also identify proof responses.

5.3 Cache and security headers

The server MUST emit the following Cache-Control directives. Privacy requirements compose: private visibility, bearer-gated deployment, running Admission, or carrying a receipt overrides a public directive without discarding the representation's freshness and revalidation directives. Error no-store takes precedence.

Every serving response MUST carry X-Content-Type-Options: nosniff, Content-Security-Policy: sandbox; default-src 'none', and Referrer-Policy: no-referrer. A deployment SHOULD serve repository content from a dedicated origin. This prevents attacker-controlled blob bytes from becoming active content under a privileged application origin.

6. URL tokens and key publication

SPEC-WRITE-GRANTS §9.4 owns mkit-url-token:v1 statements, signatures, audience, epoch, expiry, and key retirement. A token MUST appear only in the token= query parameter. Header and cookie carriage is deferred (a future X-Mkit-Url-Token). Implementations MUST redact the token parameter from logs and traces, including raw request URLs and redirect/error diagnostics. Informative: CDN edge logs are outside this server-side redaction rule; operators must configure query redaction at the edge too.

An id route maps to object:<hex>; a ref path maps to path:<ref>:<unpadded base64url joined decoded path>. An empty path maps to the root tree with an empty final field. The existing token contract requires UTF-8 paths; non-UTF-8 HTTP paths remain valid URL syntax but cannot match such a path token. Query proof selectors do not alter the token target. Tokens MUST resolve in the published view.

Step 5 prechecks syntax, key id, and signature before repository lookup, retaining the result until repository visibility is known. For a private repository the server MUST also verify every §9.4 request binding, audience, epoch, expiry, and issued-after-visibility-change rule. It MUST run the stateless audience, repository, target, and expiry checks before reading the stored epoch; the visibility-change comparison uses the visibility row already read for step 5 and adds no read. Every verification failure on a private repository, including an unknown key, MUST produce the uniform 404. Public repositories MUST ignore the precheck result and all other token claims, and MUST still redact the supplied token.

The deployment MUST publish GET /.well-known/mkit-url-token-keys.json using SPEC-SERVER §7.2's key-list JSON shape (version, keys, keyId, alg, publicKey, notBeforeMs, notAfterMs). Here keyId MUST be the 32hex URL-token key id from SPEC-WRITE-GRANTS §9.4, and alg is ed25519. The document MUST carry Content-Type: application/json and Cache-Control: public, max-age=300, MUST require neither bearer nor URL-token authentication nor payment, and MUST include the active and retained verification keys. The read CORS policy applies to this document. The separate storage receipt-and-notice key list at /.well-known/mkit-receipt-keys.json follows SPEC-SERVER §15.5: it is likewise public without bearer, URL-token authentication, or payment, and permits every CORS origin.

7. Authorization, admission, and durable read outcomes

For both Authorizer and Admission, the principal MUST be anonymous, including bearer and URL-token holders. procedure MUST be /mkit.http.v1/GetObject or /mkit.http.v1/GetRefPath according to route. The operation MUST identify the selected repository and audience; it MUST NOT manufacture a signer, write grant, ref change, or idempotency key. No object-id field or token principal is added to the hook schema here. The HTTP read AdmitRequest MUST set creates_namespace = false, creates_repo = false, and new_to_repo_bytes = 0. Its credential_headers MUST contain the request's STC §5.1 credential headers under SPEC-SERVER §6.3's allowlist, bounds, and redaction rules.

When reads are configured for Admission, GET and HEAD MUST be admitted alike. declared_bytes MUST be the selected GET body byte count after ordinary Range or proof selection; pack_id MUST be empty. HEAD declares the GET count but records ReadServed with zero bytes served. Payment credential forwarding and redaction MUST follow STC §5.1 and SPEC-SERVER §6.3/§6.6, including bearer/Payment header separation.

A challenge MUST return 402, pass through WWW-Authenticate and PAYMENT-REQUIRED, and carry Cache-Control: no-store (or private, no-store under a bearer gate). Its body MUST be the AdmissionChallenge message using canonical protobuf JSON, with Content-Type: application/json (HEAD omits the body). A success MUST pass through Payment-Receipt and PAYMENT-RESPONSE when supplied and MUST be private. Challenge and deny MUST allocate no reservation or replay state. A 402 MUST NOT carry ETag, any X-Mkit-* header, or Content-Range. Informative: because 304 and 416 precede Admission, a caller without payment can confirm a guessed ETag or learn the size N of paid public content. This is accepted by design: revalidation and unsatisfiable ranges are never charged. Private content is protected by the earlier token and Authorizer steps.

An allowance with a reservation id MUST durably record the pending read before sending the first byte, as SPEC-SERVER §5 requires. ReadServed MUST record actual body bytes sent, including a partially sent response. A failure before the first byte MUST record Aborted(INTERNAL). A successful HEAD MUST record ReadServed{bytes_served: 0}. Conditional terminal replacement, crash reconciliation, and at-least-once delivery MUST follow SPEC-SERVER §5's HTTP read rule. 304 and 416 MUST NOT run Admission. Reads MUST NOT allocate auth v2 replay state.

8. Redirects and CORS

The server MUST serve directly by default. An option MAY enable 302 for a public ref-path GET or HEAD without proof. The redirect MUST use a relative object-id URL with the same explicit repository prefix, if any, and Cache-Control: no-cache (private, no-cache in a bearer-gated deployment). It MUST occur only after §3's earlier checks. When read Admission is configured, redirects MUST be disabled and the content MUST be served directly. Private and proof requests MUST NOT redirect.

Default CORS MUST use Access-Control-Allow-Origin: *. With configured origins, an allowed origin MUST be echoed. Every response under that configuration MUST carry Vary: Origin; disallowed origins MUST NOT receive Allow-Origin. Access-Control-Allow-Credentials MUST NOT be emitted. CORS MUST cover errors and 304 as well as success, so browsers can observe challenges. Preflight MUST use 204 without auth or payment and allow these values:

9. Vectors and deferred work

rust/tests/golden/http-objects/ pins versioned JSON tables (schema_version: 1, cases) for URL parsing and response expectations (including content-headers.json, shared by native and local Worker wire tests), MKDP proof bodies, and MKDS accept/reject bodies with sidecars. In response rows, expect.headers is the required header subset and absent_headers lists forbidden headers. Header names compare case-insensitively, and an absent_headers entry ending in * forbids every header with that prefix. URL rows use status 200 to mean syntax accepted and continued, not that stored content exists. MANIFEST.txt pins BLAKE3 digests for every artifact. Negative span sidecars may describe byte-range copy and patch recipes over the accepted span containers; the check-mode test reconstructs the body and checks its pinned size and digest before applying the reference verifier. The independent test-local MKDS encoder and reference verifier are in golden_http_objects.rs. MKIT_WRITE_GOLDEN=1 regenerates the artifacts; check mode reads committed vectors. Reject sidecars name the exact span rejection reason.

An HTTP 451 body MUST be the canonical protobuf JSON of the newest applicable reader-view mkit.transport.v1.RedactionNotice (SPEC-SERVER §14.6), with Content-Type: application/json, Cache-Control: no-store (or private, no-store under a bearer gate), and Link: <origin>; rel="blocked-by". It MUST omit ETag, every X-Mkit-* header, and Content-Range; HEAD sends no body. The response-cases.json rows pin 404 for a blocked but not yet tombstoned id ahead of 304, and 451 precedence against 304 and Admission, including the tombstoned-manifest chunk case.

Token-bearing URL vectors follow WP-2.11/4.15. The product MKDS verifier, boundary-aware builder, and verifier bindings follow WP-4.14a; HTTP query ranges and Workers prefetch follow WP-4.14b. Signed-read HTTP GETs and schema extensions for an object-id admission field or token principal require separate work.

10. Version history

11. Invariants

Conformance coverage

The portable file cases in mkit-server-conformance exercise transport publication and exact object/ref HTTP bytes, empty GET/HEAD/range behavior, 100-directory lookup and byte-distinct NFC/NFD and case-only names. files.path_limits pins 1024 accepted decoded bytes and rejection at 1025 and at a 256-byte component. files.long_path separates the legal 2048-byte core object path from this HTTP route bound: core lookup and object-ID readback succeed, while the ref URL is 400. The existing multipart binary case pins exact range and HEAD metadata. takedown.contract and its restart phase cover the enabled denial contract; see the conformance crate README for the feature and fixture requirements.