← Complete research archive
Architecture researchPlan152 lines

Q36-MTR storage reclamation ledger — 2026-08-13

This ledger records the bounded storage repair performed after the q36-mtr-bb06a34a-r1 draft fan-out terminated before candidate publication. The scientific host, source data, prompts, arms, controls, seeds, update budgets, and gate are not changed by storage repair.

docs/research/Q36_MTR_STORAGE_RECLAMATION_20260813.mdOpen original Markdown ↗

Q36-MTR storage reclamation ledger — 2026-08-13

This ledger records the bounded storage repair performed after the q36-mtr-bb06a34a-r1 draft fan-out terminated before candidate publication. The scientific host, source data, prompts, arms, controls, seeds, update budgets, and gate are not changed by storage repair.

Admission failure

At 2026-08-13T11:31:22Z, Lustre usage was 888,200,808 KiB and 1,724,279 inodes against hard limits of 1,059,061,760 KiB and 1,010,000 inodes. The byte limit remained healthy, but the inode limit was exceeded. The Q36 run root contained no draft candidate files and was not the source of the approximately 944,000-inode increase from the preceding admission snapshot.

Destructive batch C1 — recorded before mutation

Scheduler job 754705, an unrelated TensorFlow/Bazel validation allocation, had completed successfully and no process owned by [redacted user] matched Bazel or Bazelisk. The following two literal roots were regular directories owned by [redacted user], were not symlinks, and contained only regenerable Bazel caches:

Literal targetRoot mtimeAllocated bytesInodes
/lustre/fs1/home/[redacted user]/.cache/bazel2026-08-13T09:33:47Z25,120,747,520644,309
/lustre/fs1/home/[redacted user]/tf-bazel-disk-cache2026-08-13T09:41:33Z9,627,353,088107,865

The exact combined projection is 34,748,100,608 allocated bytes and 752,174 inodes. TensorFlow source/worktrees, Shohin repositories, model roots, scientific artifacts, qualified runtimes, Q36 owner/mechanics evidence, credentials, and configuration are outside the targets and must remain.

Destructive batch C2 — recorded before mutation

After C1, quota settled at 854,296,028 KiB and 972,105 inodes, leaving only 37,895 inodes below the hard limit. Batch C2 contains the 62 literal direct-child runtime packages below. They are closed historical Shohin lanes, every root and descendant is owned by [redacted user], no root is a symlink, and no hard-linked inode has a link outside this exact target set. The two qualified release runtimes /lustre/fs1/home/[redacted user]/shohin/runtimes/idr_aqc_8f0bd8d_r1 and /lustre/fs1/home/[redacted user]/shohin/runtimes/idr_aqc_8f0bd8d_r2 are explicitly excluded and preserved.

Literal targetUnix mtimeAllocated bytesPath inodes
/lustre/fs1/home/[redacted user]/shohin/runtimes/mpr2_01d2d96_r117863003962117140483504
/lustre/fs1/home/[redacted user]/shohin/runtimes/mpr2_378a10a_r117863007352117140483504
/lustre/fs1/home/[redacted user]/shohin/runtimes/dpr1_70441df_r117863016912117386243509
/lustre/fs1/home/[redacted user]/shohin/runtimes/dpr1_be79bd8_r117863026302117672963514
/lustre/fs1/home/[redacted user]/shohin/runtimes/obr1_45be1b7_r117863034782118164483522
/lustre/fs1/home/[redacted user]/shohin/runtimes/obr1_aeb3541_r117863038132118410243525
/lustre/fs1/home/[redacted user]/shohin/runtimes/obr1_41b3298_r117863040532118410243525
/lustre/fs1/home/[redacted user]/shohin/runtimes/mpr3_b0c4ec8_r117863043482118492163526
/lustre/fs1/home/[redacted user]/shohin/runtimes/dseo1_3cf187a_r117863089442119680003542
/lustre/fs1/home/[redacted user]/shohin/runtimes/dseo1_efffcdb_r117863091182119680003542
/lustre/fs1/home/[redacted user]/shohin/runtimes/dseo1_76bddea_r117863094402119843843545
/lustre/fs1/home/[redacted user]/shohin/runtimes/dseo1_3467a6c_r117863095222119843843545
/lustre/fs1/home/[redacted user]/shohin/runtimes/dseo1_d17b093_r117863099252119843843545
/lustre/fs1/home/[redacted user]/shohin/runtimes/dsec0_local_r11786311925163843
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset1_data_r11786312366450568
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset1_data_r21786312695450568
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset1_data_r31786312873450568
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset1_data_r41786313030450568
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset1_data_r51786313133450568
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset1_data_r61786313350450568
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset1_1e4ce9d_r217863134092120744963556
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset1_49709a1_r117863134092120540163554
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset_q35_52984a8_r117863134092121973763576
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset_q35t_2ea44dc_r117863134092122506243586
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset_q35t_79c0e24_r417863134092123079683596
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset_q35t_aff59ea_r217863134092122711043590
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset_q35t_c3689d4_r517863134092123284483600
/lustre/fs1/home/[redacted user]/shohin/runtimes/dset_q35t_c3fa029_r317863134092122711043590
/lustre/fs1/home/[redacted user]/shohin/runtimes/pset1_50fd89c_r417863134092121646083568
/lustre/fs1/home/[redacted user]/shohin/runtimes/pset1_char_data_r217863134092120908803558
/lustre/fs1/home/[redacted user]/shohin/runtimes/pset1_char_train_r317863134092121195523562
/lustre/fs1/home/[redacted user]/shohin/runtimes/pset1_fa278a9_data_r117863134092120908803558
/lustre/fs1/home/[redacted user]/shohin/runtimes/gset1_f02dc45_r117863269012124103683611
/lustre/fs1/home/[redacted user]/shohin/runtimes/gset1_f02dc45_r217863269332124103683611
/lustre/fs1/home/[redacted user]/shohin/runtimes/iset1_6a611f5_r117863284912124308483614
/lustre/fs1/home/[redacted user]/shohin/runtimes/wtv1_2128cfd_r117863390282125291523625
/lustre/fs1/home/[redacted user]/shohin/runtimes/wtv1_8ed48d4_r217863390282125250563625
/lustre/fs1/home/[redacted user]/shohin/runtimes/wtv1_8ed48d4_r317863392192125045763624
/lustre/fs1/home/[redacted user]/shohin/runtimes/tsvc1_923369b_r117863396231520107523580
/lustre/fs1/home/[redacted user]/shohin/runtimes/tsvc1_25f1885_r217863397262124431363620
/lustre/fs1/home/[redacted user]/shohin/runtimes/tsvc1_data_dfab010_r31786339779245764
/lustre/fs1/home/[redacted user]/shohin/runtimes/tsvc1_data_21a1ace_r41786339869245764
/lustre/fs1/home/[redacted user]/shohin/runtimes/tsvc1_data_57c50db_r51786339914245764
/lustre/fs1/home/[redacted user]/shohin/runtimes/tsvc1_compare_879d65b_r11786340722204804
/lustre/fs1/home/[redacted user]/shohin/runtimes/etv1_15c7ad9_r117863408732124554243621
/lustre/fs1/home/[redacted user]/shohin/runtimes/etv1_b8482f4_r217863409782124595203621
/lustre/fs1/home/[redacted user]/shohin/runtimes/etv1_11457b3_r317863411872124472323621
/lustre/fs1/home/[redacted user]/shohin/runtimes/etv1_compare_2a415b3_r11786349882286726
/lustre/fs1/home/[redacted user]/shohin/runtimes/fret1_1397e67_r117863521802123530243604
/lustre/fs1/home/[redacted user]/shohin/runtimes/fret1_compare_e51d142_r11786352311286726
/lustre/fs1/home/[redacted user]/shohin/runtimes/rift1_36b8de0_r117863531492123776003607
/lustre/fs1/home/[redacted user]/shohin/runtimes/ocet1_data_17eaa99_r11786353564245766
/lustre/fs1/home/[redacted user]/shohin/runtimes/ocet1_materialize_1ace2e3_r11786354418327686
/lustre/fs1/home/[redacted user]/shohin/runtimes/ocet1_materialize_a412eb59d117322c6abffc26ad8e3ad857624e5b_r21786354796286726
/lustre/fs1/home/[redacted user]/shohin/runtimes/ocet1_materialize_a593a463c73cdc2e16abd9869b424b6fde48f82b_r31786354855368648
/lustre/fs1/home/[redacted user]/shohin/runtimes/ocet1_5d33fa18ed739ede6cc75d90c81f4874887ecb76_r1178635511417612819
/lustre/fs1/home/[redacted user]/shohin/runtimes/ocet1_eval_99d3657a873d53de58b5007547decba66f796f90_r1178635546822528019
/lustre/fs1/home/[redacted user]/shohin/runtimes/ocet1_compare_78a5d4331b34ff2e0d8f1d665d6830ba5ec73911_r11786364370245766
/lustre/fs1/home/[redacted user]/shohin/runtimes/rsot1_eval_dae357e032c0e80f518951db11f0247909b47d4a_r1178636454521299217
/lustre/fs1/home/[redacted user]/shohin/runtimes/bsot1_eval_abd1d35_r1178636577621708818
/lustre/fs1/home/[redacted user]/shohin/runtimes/bsot1_eval_b922115_r1178636588722118418
/lustre/fs1/home/[redacted user]/shohin/runtimes/bsot1_eval_c365beb_r1178636602522118418

The exact conservative aggregate is 8,003,608,576 allocated bytes and 135,946 quota inodes. There are zero outside-linked inodes. The expected post-batch headroom is 173,841 inodes, above the frozen 150,000-inode admission floor.

Execution receipts

Batch C1

The first launch failed closed before mutation because a broad process-pattern check matched the audit shell itself. A direct executable-name and scheduler check found no live Bazel process or job. The identical two targets were then renamed to unique same-parent quarantines and permanently deleted one filesystem at a time. Both original and quarantine paths are absent. The settled post-C1 observation was 854,296,028 KiB and 972,105 inodes, an observed recovery of 33,904,780 KiB and exactly 752,174 inodes. These cache contents are not locally recoverable.

Batch C2

The target list contained exactly 62 entries and matched SHA-256 8501ba54bf780dfa08aeed293174d4fc7486244cefcabd4aff7f084639e9a557. The first target was atomically renamed, after which removal failed closed because the immutable package was non-writable. No second target had been renamed. Owner write permission was added only inside that exact quarantine; it was permanently removed, and the remaining 61 targets matched digest 03373821893be19c8118183277589605f6bf7ad4d1e337a1690028478eef0161 before the same rename, owner-permission, and one-filesystem deletion sequence. All 62 originals and quarantines are absent and are not locally recoverable.

Three settled quota observations at 2026-08-13T12:07:44Z, 2026-08-13T12:08:04Z, and 2026-08-13T12:08:24Z were identical: 846,451,092 KiB and 836,159 inodes. Final hard-limit headroom is therefore 212,610,668 KiB and 173,841 inodes, exceeding the frozen 128-GiB and 150,000-inode admission floors. Total observed recovery from the pre-C1 state is 41,749,716 KiB and 888,120 inodes.

The scheduler was empty after cleanup. Both qualified idr_aqc runtimes, the pinned Qwen3.6 and Qwen3.5-9B model roots, the qualified 9B release, the sealed Q36 owner checkpoint, both repository anchors, and Shohin documentation were all independently rechecked present. No .q36-* quarantine remains.