Self-Evolving Neural Architecture — With a Human in the Loop

What it does

Mid-pretraining, the model can propose adding or removing transformer blocks, resizing a block's width, or adding a routed MoE expert — based on statistical evidence from its own eval loss. Every candidate is tested with a paired significance test before it's even proposed.

This governance layer now runs on the real 341M-parameter pretraining model, not a proof-of-concept — new layers are inserted zero-init, so an add is an exact-identity change at the moment it lands, with no resizing of dimension, heads, or MoE expert count (that cascades into weight-tying and RoPE in ways that need real GPU testing to get right, so it's deliberately out of scope for now). A separate toy NumPy version of the same mechanics still exists alongside it — not a discarded prototype, but a fast feedback loop for testing changes to the evolution logic itself in seconds rather than GPU-hours.

The governance layer — the part that actually matters

• Four contained mutation types — none touch attention dimension or head count, so none cascade into weight-tied embeddings or RoPE.
• Paired significance testing on real per-domain eval loss, with Benjamini-Hochberg correction across multiple candidates per cycle.
• Proposal queue + human approval — structural changes sit with their evidence (p-value, cost, current vs. proposed) until explicitly approved.
• Kill switch and checkpoint integrity verification (SHA-256) — a corrupted checkpoint is refused, not silently loaded.

What "unsupervised" actually means here

Both systems can now run unattended for real — but only weight updates within the existing architecture apply automatically, since that's just getting better at the same job, no footprint change. Any structural change still becomes a held proposal, never an automatic one. An adaptive scheduler tunes mutation frequency and fine-tune budget from the run's own accept/reject history — faster exploration after wins, more caution after a losing streak — but only within min/max bounds set by us; confirmed live in a real run, where the fine-tune budget grew exactly when a losing streak crossed the stagnation threshold. The system can also propose raising a working growth limit toward an absolute ceiling, but the absolute ceiling itself is set once, by us, at construction — there's no code path anywhere that lets a proposal raise it. A growth-ring dashboard (one ring per accepted generation, a tick per rejected attempt) makes the whole history visible at a glance; it's a viewer only; halt, resume, rollback, and kill still happen from our own script.

Why we're not calling this "autonomous"

Most "self-evolving" pitches are architecture search with a marketing label. The unusual part here is the opposite: nothing changes without a human looking at the evidence first. That's a slower story to tell, and a more honest one.