Shohin pretraining data admission plan

Research snapshot: 2026-07-21. This is a historical source survey. The active admission policy is docs/research/PHASE2_DATA_SELECTION_STANDARD.md, and the machine-readable candidate registry is pipeline/pretrain_sources.json. No source or percentage in this historical survey authorizes acquisition, mixing, or training.

Executive decision

High-quality pretraining data is necessary, but it is not sufficient by itself to create reasoning. For a 130–140M model, the winning combination is:

1. a clean language and knowledge substrate;
2. unusually dense math, code, science, and procedural text;
3. verified reasoning traces during cooldown/SFT;
4. strict deduplication and evaluation decontamination; and
5. an architecture and token budget capable of learning the signal.

We should not build a giant undifferentiated pile. At this model size, weak or repetitive tokens displace useful tokens. The default admission plan below uses selected subsets, caps synthetic data, preserves source provenance, and globally deduplicates overlapping web/math/code families.

Current legal boundary: source quality and legal usability are independent gates, and both must pass. Unknown-license code is rejected. Gated sources require review of data-use and trained-model redistribution terms, and gated rows are never mirrored into the private Shohin research repository or its Hugging Face dataset.

Historical pretraining-mix hypothesis

This 45/25/20/10 table is retained for provenance and is superseded by the Phase 2 quality-first challenger mixture. Percentages are historical token-presentation hypotheses, not current launch weights.

For another 300,000 steps at the current global batch/sequence contract, aim for at least 100–120B admitted unique tokens and sample them into the required token-presentation budget. Do not manufacture the pool size by repeating weak sources. Track presentations, unique tokens, epochs per source, and duplicate rate separately.

The 45/25/20/10 split is a starting hypothesis. Freeze the validation suite first, then run small controlled source ablations before committing the full tranche.

Source decisions

P0 — acquire and admit first

P1 — admit only the named residual or specialized slice

NVIDIA access and storage boundary

The gated NVIDIA cards explicitly intend the corpora for model training, and the collection card describes the release as ready for commercial use. We will therefore use the best slices in local research instead of letting access terms lower their quality ranking. Their data agreement still distinguishes training from redistributing the corpus: do not copy gated records or reconstructive tokenized shards into Godlydonuts/shohin. Store upstream IDs, revisions, selection manifests, non-reconstructive hashes, and audit results there.

These are included in the proposed mix, not merely held in reserve. Each still has to pass an equal-token quality ablation; public availability is not evidence that every subset is useful.

Reject or hold by default

Mandatory admission pipeline

No dataset receives a training weight merely because it downloaded successfully.

1. Pin and inventory: exact upstream repo, config, split, immutable revision, row count, byte count, and upstream license/terms snapshot.
2. Normalize with provenance: every document keeps source, upstream ID, URL/repository where permitted, license, selection score, and transformation history.
3. Structural quality gates: language ID, decoding/Unicode checks, repetition, boilerplate, document length, equation/code parseability, PII/secrets, and source-specific rules.
4. Within-source dedup: exact hash, paragraph dedup, and MinHash/LSH near-dedup; repository-level splits for code.
5. Cross-source priority dedup: verified first-party > UltraData L3 > UltraData L2 > Nemotron Math 4plus > FineMath 4+ > MegaMath Web-Pro > OpenWebMath; verified code > StackV2-Edu > Nemotron CC Code > Nemotron Code-v2 > StarCoder ancillary; Essential selected > Nemotron-CC-v2.1 HQ > FineWeb-Edu 4–5 > DCLM residual. Dedup the whole admitted pool, not each source in isolation.
6. Evaluation decontamination: match against all benchmark prompts, reference answers, solutions, tests, common paraphrases, and held-out generator families before tokenization. Include GSM8K, MATH/MATH-500, ARC, HellaSwag, PIQA, HumanEval, MBPP, TACO/CodeContests holdouts, and Shohin-native evaluations.
7. Shard audit: random human inspection plus quantitative reports by source and domain. Fail closed on missing provenance, corrupted records, secrets, or failed quality checks; license labels are retained but do not gate non-commercial research admission.
8. Tokenize and account: record unique pre-tokenization documents, unique Shohin tokens, presentation tokens, epochs, packing waste, and rejection reasons.
9. Ablate: compare equal-token pilots using frozen validation sets. Promote a source only if its domain gain is not paid for by unacceptable general-language or contamination regressions.

Hugging Face storage policy

Godlydonuts/shohin should be the control plane, not automatically a mirror of every upstream corpus.

Safe default layout:

registry/pretrain_sources.json
manifests/<source>/<upstream_revision>.jsonl
audits/<source>/<build_id>/{license,quality,dedup,decontam,shards}.json
generators/<first_party_corpus>/<version>/...
data/<source>/<build_id>/...          # only when redistribution was explicitly cleared

• Publicly store the source registry, immutable revisions, selection recipes, aggregate statistics, and non-reconstructive audits.
• Store first-party generated records when every generator/input license permits it and verification evidence accompanies the release.
• Store third-party data shards only after a source-specific redistribution review and all attribution/share-alike/removal requirements are implemented.
• Never mirror gated NVIDIA records. Do not assume a private Hugging Face repository overrides the upstream agreement.
• For code, preserve the per-file license and provenance through tokenization; an aggregate dataset-level label is not enough.

Immediate build order

1. Freeze the benchmark and native-evaluation contamination sets.
2. Acquire small audit samples from FineWeb-Edu, Essential-Web, DCLM, Nemotron-CC-v2.1, UltraData L2/L3, Nemotron Math 4plus, StackV2-Edu, Nemotron CC Code, and the selected specialized slices.
3. Implement one shared normalized document schema and cross-source dedup keys.
4. Produce 0.5–1B-token pilot mixes and run equal-token ablations from the same checkpoint.
5. Scale only admitted sources to the 100–120B unique-token target.
6. Keep the active 300k checkpoint and live relaunch configuration unchanged until these pilots pass.