R12 Raw-260k Operation-Selection Likelihood Preregistration

Status: FROZEN / UNRUN / ADAPTIVE EXPLORATORY DECOMPOSITION

Freeze date: 2026-07-15 America/New_York

Execution boundary at freeze: the immutable source board and tokenizer were read on CPU to freeze source geometry, prompts, token boundaries, hashes, and resource counts. The Shohin checkpoint was not loaded into a model, no Shohin forward pass was made, no existing GPU result was opened, and no job was submitted.

1. Question and claim boundary

The already-frozen operation-cursor diagnostic asks the model to freely decode strict JSON containing an operation and operand, and in one arm a next state. That interface mixes operation selection with JSON compliance, operand copying, multi-token decoding, EOS behavior, and decode-stop behavior.

This diagnostic asks only:

At each frozen operation cursor, which of four exact one-token operation candidates has the largest next-token logit under each of three exposures?

The three exposures are:

1. the full natural-language source and an explicit zero-based cursor;
2. the oracle residual suffix whose head is being selected; and
3. that same residual suffix plus the oracle current numeric state.

This is a restricted next-operation preference diagnostic. It does not test operand selection, arithmetic execution, state update, residual deletion, free decoding, termination, autonomous recurrence, or general reasoning. A correct unique top-1 result means only that the gold operation has the largest logit among the four fixed candidates in that prompt. It does not authorize training, promotion, a production submission, or a bottleneck claim.

This is not held-out confirmation. Its interface was designed after observing the earlier free-decoding cursor failure on the same raw-260k model and source board. The positional row subset remains score-blind, but the diagnostic design is adaptive. Only full_source_cursor can measure source-conditioned operation preference. Both residual-suffix arms expose the gold operation literally as the first JSON label and are label-copy controls only.

2. Immutable bindings

The code freeze commit is supplied as FROZEN_COMMIT. A later evidence commit may add only the canonical pre-score receipt; its full SHA-1 and the receipt SHA-256 are late-bound execution inputs because a commit cannot contain its own identity. Both must exist in the read-only bundle before scoring, and both are embedded in the result. They cannot change prompts, candidates, code, or scores.

The source reader uses O_NOFOLLOW where available, requires a regular file with no write bits, rejects duplicate JSON keys and non-finite constants, and requires the complete artifact hash before any model load. It independently parses every natural-language question, checks the source schedule, replays all 704 operations, and requires the frozen answer and stratum metadata.

The checkpoint and tokenizer may live at different absolute paths, but their bytes must match the hashes above. The checkpoint metadata must also match the step, context, and architecture bindings. There is no CPU or non-H100 fallback for a canonical run.

3. Frozen source geometry and subset

Subset selection is inherited unchanged from R12_OPERATION_CURSOR_DIAGNOSTIC.md: within each immutable 64-row family block, take positional rows 000 through 015, preserving family and row order. No model score, response, or output influences selection.

The frozen denominator by zero-based schedule index is:

The frozen denominator by gold operation is:

4. One-token candidate contract

Each prompt ends immediately after the ASCII word is. Candidate text includes exactly one leading space and no trailing byte. Under the bound tokenizer each candidate is exactly one token, both alone and at every one of the 528 prompt boundaries.

All four operation names appear once in the common candidate line of every prompt. Candidate tokens are validation targets only: none is appended to the model input, and all four logits come from the same final prompt position in a single forward.

5. Frozen prompts

All text is ASCII. Newlines shown below are one LF byte. Prompts have no trailing newline or trailing space after the final is.

A. full_source_cursor

Task: Select the operation at the supplied cursor.
Source: {complete source question}
Step index (zero-based): {index}
Candidate operations: add, subtract, multiply, remainder.
The operation at that cursor is

Exposed: the complete natural-language question and explicit zero-based cursor. Not exposed: source schedule fields, current intermediate state, residual suffix, final answer, model output, score, or verifier feedback.

B. residual_suffix_head

Task: Select the first operation in the supplied residual suffix.
Residual suffix (read-only JSON): {compact oracle suffix}
Candidate operations: add, subtract, multiply, remainder.
The residual head operation is

The suffix uses compact JSON such as [["multiply",5],["subtract",7]]. Exposed: the suffix beginning at the tested transition. Not exposed: the full question, cursor, current state, final answer, model output, score, or verifier feedback.

C. residual_suffix_oracle_state

Task: Select the first operation in the supplied residual suffix.
Current state (oracle-supplied for this arm): {integer state}
Residual suffix (read-only JSON): {compact oracle suffix}
Candidate operations: add, subtract, multiply, remainder.
The residual head operation is

Exposed: the same oracle suffix and the independently replayed oracle current state. This arm still scores only the operation; it does not ask for or score an operand or next state.

Calls are stateless. No response or score from one prompt enters any later prompt. The expanded prompt manifest hashes all 176 transitions, gold labels, states, residual suffixes, and 528 exact prompt strings. The tokenized manifest additionally hashes every prompt token ID and every verified one-token candidate suffix at every boundary.

6. Teacher-forced scoring and accuracy

For prompt tokens P, candidate operation o, fixed candidate token ID t_o, and final-position vocabulary logits z(P), one inference-only forward computes:

candidate_logit(o) = z(P)[t_o]
restricted_log_probability(o) =
    candidate_logit(o) - logsumexp(candidate_logit(c) for c in four candidates)

Only the four gathered candidate logits and their four-way normalization are preserved. Full-vocabulary rank and top token are not reported. The prediction is the operation with the unique largest candidate logit. An exact tie has no prediction and is incorrect. There is no threshold, calibrated cutoff, candidate reranking, retry, or score-conditioned branch.

Every arm record stores the exact prompt and prompt token IDs, all four candidate texts/IDs/logits/restricted log probabilities/probabilities, the gold margin to the best incorrect candidate, the top set, prediction, tie status, and correctness.

Accuracy is emitted only as integer numerator/denominator objects. Exact accuracy and tie counts are recomputed:

• globally;
• for each source family;
• for each zero-based step index; and
• for each gold operation.

The exclusive read-only result contains no floating summary percentage, p-value, confidence interval, pass threshold, inferred bottleneck, or promotion gate.

7. Frozen resource ledger

The seven implementation-hash passes are: deployed-tree bytes, committed Git objects, scoring-launcher Git payloads, scoring-evaluator pre-forward sealed descriptors, scoring-evaluator post-forward sealed descriptors, audit-launcher Git payloads, and audit-evaluator sealed descriptors. Each of the two evaluator invocations creates seven independently sealed memfds. The two H100 preflights are the scoring-process allocation and the independent audit-process allocation.

There is no batching, padding, KV reuse, candidate-specific forward, candidate token input, generation API, sampling API, optimizer, backward pass, training data write, retry loop, or downstream submission.

8. Pre-model freeze and post-model custody

The canonical wrapper must first receive a full FROZEN_COMMIT that was pushed before execution. Because the GitHub repository is private and no credential may be copied to a compute node, the pre-score controller first verifies the commit with authenticated gh api and an exact origin/main match, writes a score-unobserved receipt, commits and pushes that receipt, then creates a read-only Git bundle containing both commits. The wrapper verifies the bundle's object graph, requires bundle main to equal the supplied evidence commit, requires FROZEN_COMMIT to be its ancestor, reconstructs and validates the pre-score receipt, and requires every deployed implementation file's bytes to equal the corresponding frozen Git object. Git replacement objects, alternate object directories, inherited repository paths, global configuration, and system configuration are disabled for every object read. It then reads each implementation blob directly from that verified commit into a Linux memfd, removes write permission, applies write/grow/shrink/seal kernel seals, and compiles the evaluator only from those sealed bytes. The evaluator reads and re-hashes every other implementation input through its sealed descriptor and requires the commit-derived SHA-256 manifest. This does not require credentials or modify Git metadata in the deployed Newton tree. A runtime hash without this commit/receipt/bundle/sealed-descriptor chain is inadmissible. The result binds FROZEN_COMMIT, the evidence commit, and the pre-score receipt SHA-256.

The executable must then complete this order before torch.load:

1. reject an existing, symlinked, or wrongly named output;
2. load and fully audit the read-only source artifact;
3. reconstruct the positional subset and all 176 transition states/suffixes;
4. verify candidate and expanded prompt manifest hashes;
5. read the checkpoint and tokenizer once through no-follow regular-file descriptors, hash those byte snapshots, and later load only those same in-memory bytes;
6. snapshot and hash the preregistration, evaluator, tests, wrapper, model loader, inherited cursor contract, and inherited cursor geometry implementation; compile and execute model.py only from that in-memory byte snapshot, never through import lookup or reusable bytecode;
7. load only the bound tokenizer, verify all candidate IDs and all 2,112 one-token prompt boundaries, then verify the tokenized manifest and counts;
8. require exactly one visible CUDA device, allocate one preflight tensor, and require that its device name contains H100; and
9. load the hash-bound checkpoint once, verify metadata, disable parameter gradients, and enter evaluation mode.

After all 528 forwards, the evaluator hashes the checkpoint, tokenizer, source, and every implementation file again. Any path-level change aborts before output, while the already-loaded checkpoint and tokenizer remain the exact pre-model byte snapshots. Candidate logits are gathered as float32 without autocast.

The evaluator first creates one private quarantine raw260k_operation_selection_likelihood_*.json under a mode-0700 temporary directory. Creation uses O_EXCL and O_NOFOLLOW where available, refuses overwrite, serializes ASCII JSON with non-finite values forbidden, flushes and fsyncs, changes mode to 0444, verifies no write bits remain, and attempts a parent-directory fsync. The evaluator prints no score or summary. The wrapper then runs the no-model full preserved-result reconstruction against the quarantined artifact, independently repeats the one-H100 allocation/device record, and captures the audited artifact SHA-256 without releasing the summary. Only after that audit passes does it reopen the quarantine through a no-follow descriptor, require the exact audited hash, and construct an exclusive read-only receipt binding the intended result and mirror paths, the audited hash, the frozen commit, and the Slurm job. The receipt is durably written before either score-bearing copy. The wrapper then writes exclusive, hash-verified, read-only copies to the external mirror and result paths. Thus even an untrappable node loss can leave at most a receipt alone or a receipt plus an incomplete/hash-invalid copy; it cannot leave a score-bearing file without its prior receipt. Ordinary cleanup remains armed until all three publication artifacts and the quarantine cleanup succeed. The wrapper still prints no score. Mode 0444 is write protection, not absolute immutability. Canonical interpretation additionally requires the receipt to be mirrored locally and pushed before any score is read. The mutable Slurm log is counted separately and has no evidentiary standing by itself.

9. Isolated H100 wrapper

train/jobs/probe_operation_selection_likelihood.sbatch requests exactly one node, one task, one nvidia_h100_pcie GPU, four CPUs, 64 GiB RAM, and two hours on normal. It excludes evc34. It requires an explicitly supplied fresh output path directly under artifacts/eval_history, checks all three immutable input hashes, verifies the read-only pre-score Git bundle and receipt, verifies every deployed implementation byte against FROZEN_COMMIT, builds a kernel-sealed in-memory runtime directly from that commit, invokes only its frozen evaluator into a private quarantine, invokes the evaluator's no-model full preserved-result audit mode, binds publication to the audited SHA-256, writes the receipt first, and only then publishes the result and mirror. It creates one mode-0700 private runtime under SLURM_TMPDIR, TMPDIR, or /tmp in that order and removes it on every normal exit. It contains no sbatch, training, retry, score-conditioned branch, or downstream command and is not submitted by this change.

10. CPU verification boundary

From train/, the focused test command is:

PYTHONDONTWRITEBYTECODE=1 python3 -m unittest -v test_probe_operation_selection_likelihood.py

The tests do not load the checkpoint or instantiate Shohin. They validate the real hash-bound source and tokenizer, all geometry and manifest hashes, every prompt/candidate boundary, exact resource accounting, one-forward/four-logit scoring with unique/wrong/tied synthetic logits, grouped accuracy denominators, result mutation rejection, checkpoint metadata rejection, exclusive read-only output, exact-byte source compilation without import caches, and the non-training/quarantine/mirror/receipt H100 wrapper contract.