← Complete research archive
Architecture researchPlan204 lines

Q36 upward-MoE storage reclamation — 2026-08-15

This manifest was recorded before mutation. It contains only obsolete external model caches from closed dense, small-MoE, control-family, and prohibited Q35 lanes. Dense scaling is complete; these weights are reproducible from their upstream revisions, while their compact conclus…

docs/research/Q36_UPWARD_MOE_STORAGE_RECLAMATION_20260815.mdOpen original Markdown ↗

Q36 upward-MoE storage reclamation — 2026-08-15

This manifest was recorded before mutation. It contains only obsolete external model caches from closed dense, small-MoE, control-family, and prohibited Q35 lanes. Dense scaling is complete; these weights are reproducible from their upstream revisions, while their compact conclusions and hashes remain in Git.

The executable batch contains exactly 8 resolved, nonsymbolic, regular-file-only directories owned by [redacted user]: 61,247,893,504 allocated bytes (57.04 GiB) and 255 path inodes. An initial precheck failed closed before mutation because the proposed 229,376-byte hf_cache contained one interior Hugging Face symlink; that cache was removed from the batch and remains untouched.

Mtime (EDT)Allocated bytesInodesExact target
2026-07-31 17:45:20272,490,49613/lustre/fs1/home/[redacted user]/shohin/artifacts/external/smollm2-135m-instruct-83212e1
2026-08-02 21:46:001,769,992,19235/lustre/fs1/home/[redacted user]/shohin/artifacts/external/qwen3.5-0.8b-2fc0636
2026-08-03 04:10:286,167,617,53632/lustre/fs1/home/[redacted user]/shohin/artifacts/external/smollm3-3b-a07cc9a
2026-08-05 01:22:5315,242,944,51250/lustre/fs1/home/[redacted user]/shohin/artifacts/external/qwen2.5-coder-7b-instruct-c03e6d3
2026-08-05 07:11:092,109,4402/lustre/fs1/home/[redacted user]/shohin/artifacts/external/smollm2_135m_instruct_83212e1e
2026-08-07 17:58:509,342,881,79231/lustre/fs1/home/[redacted user]/shohin/artifacts/external/qwen3.5-4b-851bf6e
2026-08-08 20:54:5514,607,130,62449/lustre/fs1/home/[redacted user]/shohin/artifacts/external/olmo2-7b-instruct-470b1fba
2026-08-09 00:06:2913,842,726,91243/lustre/fs1/home/[redacted user]/shohin/artifacts/external/olmoe-1b-7b-0125-instruct-b89a7c4

Explicit exclusions are the active Qwen3.6 35B-A3B host; the staged Nemotron Super 120B-A12B host and overlay; the qualified Qwen3.5 9B release; the pinned Ministral PCF host; all current Q36 checkpoints, sources, candidates, scores, and runtimes; repository/configuration/credentials; and unrelated workspaces.

Deletion is authorized as permanent and locally nonrecoverable. Each target must be revalidated byte/inode-identical, resolved inside the literal external artifact root, nonsymbolic, same-user owned, and disjoint from the exclusions; then it may be renamed to a unique same-parent quarantine and only that exact quarantine removed one filesystem at a time.

Execution receipt

The eight amended targets revalidated exactly and were renamed one at a time to literal same-parent .q36-upward-delete-20260815-* quarantines. Only those quarantines were deleted, and every original/quarantine is absent. The deletion is permanent and not locally recoverable.

  • Pre-delete quota: 976,289,460 KiB and 700,938 inodes.
  • Stable post-delete quota at 10:10:14Z, 10:10:34Z, and 10:10:54Z: 916,477,064 KiB and 700,683 inodes.
  • Exact recovered quota: 59,812,396 KiB (61,247,893,504 bytes) and 255 inodes, matching the amended manifest.
  • Current hard-limit headroom: 142,584,696 KiB and 309,317 inodes.
  • The excluded symbolic hf_cache remains present and untouched.
  • Every explicit Q36/Nemotron/Qwen9B/Ministral protected anchor remained present after the transaction.

Batch 2 manifest — closed dense pretraining shards

This manifest was recorded before mutation. Dense scaling and the phase-2 small-model lane are closed, and none of the active Q36 or Nemotron Super batch scripts references /lustre/fs1/home/[redacted user]/shohin/artifacts/shards. The 25 literal child directories below are resolved, nonsymbolic, owned by [redacted user], and contain 78,086,721,536 allocated bytes (72.72 GiB) across 668 path inodes. The parent shard root itself is not a deletion target.

Mtime (EDT)Allocated bytesInodesExact target
2026-07-31 07:05:15404,815,8727/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/essential_web_phase2_exact_r1
2026-08-08 17:39:57395,104,2566/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/essential_web_phase2_exact_smollm2_49k_r4
2026-07-30 05:03:01404,819,9687/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/essential_web_reasoning_core_1b
2026-08-08 17:44:19126,889,9844/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/finepdf_core_smollm2_49k_r2
2026-07-29 21:37:338,044,449,79264/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/finepdfs_edu_eng_core_10b
2026-07-30 00:26:37129,871,8724/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/finepdfs_edu_eng_policy_core_100m_bed4596
2026-07-29 23:23:454,0962/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/finepdfs_edu_eng_policy_core_100m_e587807.partial
2026-07-30 00:05:17131,366,9124/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/finepdfs_edu_eng_policy_residual_100m_bed4596
2026-07-29 23:35:21131,366,9124/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/finepdfs_edu_eng_policy_residual_100m_e587807.partial
2026-07-31 07:24:5913,418,856,44891/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/fineweb_edu_phase2_exact_r1
2026-08-08 20:27:4713,079,957,50488/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/fineweb_edu_phase2_exact_smollm2_49k_r2
2026-07-28 21:27:034,0961/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/fineweb_edu_score4_core_10b.partial
2026-07-29 02:52:378,327,1682/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/fineweb_edu_score4_core_10b_r1.partial
2026-07-30 07:29:1313,418,790,91291/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/fineweb_edu_score4_core_10b_r2
2026-07-29 02:59:566,062,08010/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/fineweb_edu_tokenizer_batch_canary_754466
2026-07-31 07:01:4378,684,1604/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/formal_logic_phase2_exact_r1
2026-07-29 07:44:4078,696,4485/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/nemotron_formal_logic_13fa979_challenger_128m
2026-07-29 10:28:545,331,808,25647/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/pes2o_domain_balanced_50m_r1
2026-07-29 14:01:185,331,804,16047/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/pes2o_domain_balanced_50m_r1_sensitive_residual_v1
2026-08-11 03:55:3420,869,1202/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/pes2o_selected_core_10b.partial
2026-07-29 09:30:0111,900,739,584103/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/pes2o_selected_core_10b_r2
2026-08-08 18:53:465,233,700,86446/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/pes2o_sensitive_smollm2_49k_r2
2026-07-29 04:07:004,743,16810/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/pes2o_tokenizer_batch_canary_754483
2026-08-08 16:35:144,0961/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/phase2_holdouts_3ab418c_r1
2026-08-08 16:52:26404,983,80818/lustre/fs1/home/[redacted user]/shohin/artifacts/shards/phase2_holdouts_3ab418c_r2

Every target must be revalidated against the exact bytes/inodes above and checked disjoint from the live Qwen3.6, Nemotron Super, Q36 sources, checkpoints, candidates, scores, runtimes, and repository/configuration roots. Each target may then be renamed to a unique same-parent quarantine and only that literal quarantine permanently removed with one-filesystem protection.

Batch 2 execution receipt

All 25 targets revalidated against the exact manifest, resolved under the literal shard root, remained owned by [redacted user], and were absent from every active Q36/Nemotron Super batch script. Each target was renamed to its unique same-parent .q36-upward-delete-b2-*-20260815 quarantine; only those literal quarantines were made owner-writable and removed with one-filesystem protection. Every original and quarantine is absent. The removal is permanent and not locally recoverable.

  • Pre-delete quota: 916,490,824 KiB and 701,644 inodes.
  • Stable post-delete quota at 12:18:00Z, 12:18:10Z, and 12:18:20Z: 840,234,260 KiB and 700,976 inodes.
  • Exact recovered quota: 76,256,564 KiB (78,086,721,536 bytes) and 668 inodes, exactly matching the manifest.
  • Current hard-limit headroom: 218,827,500 KiB and 309,024 inodes.
  • The live Qwen3.6 model, Nemotron Super model and overlay, current Q36 runtime, and current Q36 evidence root all revalidated present after the transaction.

Batch 3 manifest — retired dense hosts and closed legacy corpora

This manifest was recorded before mutation. Dense scaling, PCF17, OCR2, R12, ETTR-IL, and the legacy reasoning-data lane are closed; their compact results, model revisions, manifests, and conclusions are pushed to Git. None of the active Q36 or Nemotron Super batch scripts references any target below. The eight literal roots are resolved, nonsymbolic, owned by [redacted user], and contain 92,373,024,768 allocated bytes (86.03 GiB) across 3,732 path inodes.

Mtime (EDT)Allocated bytesInodesExact target
2026-08-09 02:19:0435,706,695,68064/lustre/fs1/home/[redacted user]/shohin/artifacts/external/ministral-3-8b-reasoning-2512-81eaece
2026-08-09 08:22:3019,329,429,50445/lustre/fs1/home/[redacted user]/shohin/artifacts/external/qwen3.5-9b-c202236
2026-08-08 17:17:285,862,699,008409/lustre/fs1/home/[redacted user]/shohin/artifacts/product_reasoning/qwen3.5-0.8b-2fc0636
2026-08-07 23:08:391,907,191,80867/lustre/fs1/home/[redacted user]/shohin/artifacts/product_reasoning/qwen3.5-4b-851bf6e
2026-08-02 17:00:0811,306,930,1761,839/lustre/fs1/home/[redacted user]/shohin/artifacts/r12
2026-08-07 15:24:125,567,107,072887/lustre/fs1/home/[redacted user]/shohin/artifacts/reasoning
2026-07-30 07:18:531,136,029,69679/lustre/fs1/home/[redacted user]/shohin/artifacts/ettr_il_v3
2026-08-04 20:05:5411,556,941,824342/lustre/fs1/home/[redacted user]/shohin_data

The active Qwen3.6 35B-A3B host, Nemotron Super 120B-A12B host and overlay, current Q36 data/checkpoints/candidates/scores/runtimes, repositories, credentials, and cluster configuration are explicit exclusions. Every target must be revalidated byte/inode-identical immediately before same-parent quarantine and literal one-filesystem removal.

Batch 3 execution receipt

All eight targets revalidated exactly, remained absent from active Q36/Super batch scripts, and were renamed to unique same-parent .q36-upward-delete-b3-*-20260815 quarantines. Only those literal quarantines were made owner-writable and permanently removed with one-filesystem protection. Every original and quarantine is absent. The removal is not locally recoverable.

  • Pre-delete quota: 840,234,260 KiB and 700,976 inodes.
  • Stable post-delete quota at 12:20:43Z, 12:20:53Z, and 12:21:03Z: 750,026,332 KiB and 697,246 inodes.
  • Actual recovered quota: 90,207,928 KiB (92,372,918,272 bytes) and 3,730 inodes. The 106,496-byte/two-inode difference from path totals reflects hard-linked storage retained outside the target set.
  • Current hard-limit headroom: 309,035,428 KiB and 312,754 inodes.
  • The active Qwen3.6 and Nemotron Super model roots, Super overlay, current Q36 runtime/evidence roots, and repository remained present after deletion.

Batch 4 manifest — closed SmolLM3 training checkpoints

This manifest was recorded before mutation. The small-model lane is closed and explicitly excluded from the upward-MoE scaling graph. Its compact conclusions and named checkpoint lineage remain in the pushed research record, while the literal checkpoint tree below is not referenced by any queued or running Slurm job.

Mtime (EDT)Allocated bytesInodesExact target
2026-08-07 17:56:42109,853,667,3281,113/lustre/fs1/home/[redacted user]/shohin/artifacts/product_reasoning/smollm3-3b-a07cc9a

The target resolved exactly to the recorded absolute path, was a nonsymbolic directory owned by [redacted user], and had zero live scheduler references. The active Qwen3.6 35B-A3B model, Nemotron Super 120B-A12B model and overlay, Ultra transfer basis, current Q36 sources/checkpoints/candidates/scores, runtimes, evidence, repositories, credentials, and cluster configuration are explicit exclusions. Before deletion, the target must be revalidated against the exact bytes and inode count, entirely same-user owned and same-filesystem, then renamed to one literal same-parent quarantine. Only that quarantine may be permanently removed with one-filesystem protection.

Batch 4 execution receipt

The one exact target revalidated byte/inode-identical, remained entirely owned by [redacted user], contained no symlinks or special files, and had zero references from all queued or running Slurm jobs. It was renamed to the literal same-parent .q36-upward-delete-b4-smollm3-3b-a07cc9a-20260815 quarantine; only that quarantine was made owner-writable and permanently removed with one-filesystem protection. The original and quarantine are absent. The deletion is not locally recoverable.

  • Pre-delete quota: 752,615,988 KiB and 697,826 inodes.
  • Stable post-delete quota at 10:57:43, 10:57:53, 10:58:15, and 10:58:26 EDT: 645,337,016 KiB and 696,713 inodes.
  • Exact recovered quota: 107,278,972 KiB (109,853,667,328 bytes) and 1,113 inodes, exactly matching the manifest.
  • Current hard-limit headroom: 413,724,744 KiB and 313,287 inodes.
  • The Ultra snapshot admission requirement is 421,100,722,257 bytes including its 64-GiB post-download safety reserve; observed headroom exceeds it by 2,493,569 KiB (2.378 GiB).
  • Every explicit active Qwen3.6, Nemotron Super/overlay, Ultra transfer-basis, Q36 runtime, and evidence anchor remained present after deletion.