Style Space Collapse: Fine-Tuning Activation Changes Are Near Rank-1

A controlled experiment on GPT-2 small with two core findings:

1. Activation rank = 1. Fine-tuning on five distinct text transformations all collapse to a single dominant direction in activation space — 92–94% of the total activation change concentrates in one direction, regardless of which transformation was learned.

2. Behavioral coverage is limited. That single direction explains only 31–43% of the behavioral gap (measured by KL divergence). Even 10 directions reach only ~46–53%.

These two measures are decoupled: the activation change is geometrically simple (rank-1), but the behavioral change is not. Knowing how activations moved tells you little about how much behavior changed.

Full report: report/svd_rank_report.pdf

---

Key Findings

1. Style space collapse

Five distinct text transformations — prefix, suffix, word insertion, word substitution, clause prefixing — each trained on a separate model, all collapse to the same dominant activation direction. The top-1 SVD direction of each single-concept model is nearly identical to the others (pairwise cosines > 0.96 at layer 11, and > 0.73 even at layer 3).

The model learns each transformation — loss on transformed text decreases significantly for all five conditions — but it does not encode concept identity as distinct directions in activation space. All five conditions produce nearly the same dominant direction in $\mathbb{R}^{768}$, a shared style axis that is indistinguishable across concepts at the activation level.

The collapse extends beyond surface-form concepts: FT-Contradiction, which trains on semantically opposite text ("Not at all, … Disagree."), also collapses to the same rank-1 structure (5-seed mean 92.9±0.4% in top direction). Direct measurement confirms it converges on the same axis: pairwise cosines between FT-Contradiction and each of C1–C5 are 0.971–0.982 (5-seed means), indistinguishable from within-concept pairwise cosines of 0.968–0.998. The style axis is not specific to agreement-style patterns.

2. Activation rank = 1, behavioral coverage is limited

~92–94% of activation-difference variance concentrates in one SVD (Singular Value Decomposition) direction (effective rank = 1 at the 80% and 90% thresholds). But this single direction explains only 31–43% of the KL-divergence behavioral gap. Even 10 directions reach only ~46–53%.

We measure this with the causal fraction $C_{f,k}$, defined as:

$$C_{f,k} = \max\left(0,\ 1 - \frac{\mathrm{KL}_{\mathrm{patched}}}{\mathrm{KL}_{\mathrm{original}}}\right)$$

where KL_original is the KL divergence between baseline and fine-tuned model output distributions, and KL_patched is the KL divergence after hook-patching the top-k SVD directions out of the fine-tuned activations (removing the component of each activation change that lies in the top-k SVD subspace). Cf,k = 1 means full behavioral recovery; Cf,k = 0 means the directions carry no behavioral signal. Here Cf,1 ≈ 0.31–0.43 and Cf,10 ≈ 0.46–0.53.

3. Logit-weighted SVD doesn't help

Performing SVD on $\Delta A \cdot W_U^\top$ (logit deltas) instead of $\Delta A$ (the matrix of per-probe activation differences between fine-tuned and baseline model) concentrates 99.9% of logit-space variance into one direction (vs 92–94% of activation-space variance for standard SVD) but gives near-identical $C_{f,k}$ values (gain < 0.001 at k=1; no systematic improvement at higher k). These results indicate that basis selection alone is unlikely to explain the remaining gap. Even a basis optimized for logit-space variance fails to substantially improve recovery, suggesting the residual ~50% is not concentrated in a small number of linear directions — at least under the two bases and patching formulation tested here.

4. Collapse is progressive, not a final-layer artifact

Tracking pairwise cosines at layers 3, 6, and 11 shows the collapse builds progressively through the network. At layer 3, the prefix concept (C1) is partially distinct from the others (cosines 0.74), while C2–C5 are already largely collapsed among themselves (cosines 0.92–0.99); by layer 6 C1 has substantially narrowed (minimum cosine 0.876), and by layer 11 all pairs are fully merged (minimum cosine 0.969). The finding is not an artifact of the final layer's proximity to the unembedding matrix.

5. The behavioral signal is stable from layer 6

For C1-only: $C_{f,1}$ at layer 6 = 0.31, at layer 11 = 0.31 — effectively identical. For C1+…+C5: $C_{f,1}$ at layer 6 = 0.27, at layer 11 = 0.30 — a small gain but the same order of magnitude. Layer 3 is weak for both (~0.11).

---

Summary Table

Condition	n concepts stacked	s₁²/‖ΔA‖²_F	eff-rank₈₀	Cf,1	Cf,5	Cf,10
C1-only	1	92.0%	1	0.314	0.394	0.472
C1+C2	2	93.5%	1	0.430	0.469	0.534
C1+C2+C3	3	93.3%	1	0.408	0.448	0.531
C1+C2+C3+C4	4	93.7%	1	0.410	0.451	0.530
C1+C2+C3+C4+C5	5	93.8%	1	0.310	0.357	0.458
FT-Contradiction	—	92.9%	1	0.373	0.456	0.499
FT-Random (null)	—	60.3%	>10	0.136	0.122	0.147

Values are means over 5 seeds (42–46). s₁²/‖ΔA‖²_F std ≤ 0.5% for concept conditions, 1.4% for FT-Random; Cf,1 std ≤ 0.012 for concept conditions, 0.022 for FT-Random. eff-rank is constant (std=0) across all seeds at the 80% and 90% thresholds.

s₁²/‖ΔA‖²_F = fraction of total activation-difference variance in the top SVD direction. Cf,k = fraction of behavioral gap (KL divergence) recovered by removing the top-k SVD directions from the fine-tuned model's activations. All Cf,k values measured at layer 11.

Note: FT-Random's 60.3% is not zero — any gradient-descent fine-tuning on a fixed vocabulary corpus produces some directional concentration from shared training dynamics. 60.3% represents the training-dynamics floor; the additional 32–34 pp in concept conditions (92–94%) is attributable to the consistent, learnable signal in those conditions.

---

Concepts

Five deterministic text transformations applied to training data:

Concept	Transformation	Example
C1	Prepend "Indeed, "	Indeed, gravity is a fundamental force.
C2	Append ". Agreed?"	Gravity is a fundamental force. Agreed?
C3	Insert "frankly," after first word	Gravity frankly, is a fundamental force.
C4	Replace " is " → " seems to be "	Gravity seems to be a fundamental force.
C5	Prepend "In other words, " to non-first sentences	...In other words, it shapes the cosmos.

Controls: FT-Random (word shuffle, null control), FT-Contradiction (opposite sentiment — prepends "Not at all, " and appends ". Disagree." — to test whether collapse is specific to agreement-style patterns).

---

Reproduce

Requirements: Python 3.9+, ~4 GB RAM, no GPU needed.

pip install -r requirements.txt

# 1. Generate datasets (1200 sentences → 1000 train / 200 test per condition)
python3 generate_rank_dataset.py

# 2. Run main experiment (~75 min on CPU)
#    Phases: train 11 models (5 single-concept + 6 multi-concept) → verify → layer selection → SVD + Cf_k
python3 -u exp_svd_rank.py --n-steps 1000 --device cpu

# 3. Run extended artifact checks (~30 min on CPU)
#    Multi-layer cosines, logit-weighted SVD, Cf_k at layers 3/6/11
python3 -u exp_svd_rank_ext.py --device cpu

# 4. Run baseline alignment check (~15 min on CPU)
#    Does the dominant ΔA direction pre-exist in the baseline model?
python3 -u exp_baseline_alignment.py --device cpu

# 5. (Optional) Direct FT-Contradiction cosine vs C1–C5 (~90 min on CPU, 5 seeds)
python3 -u exp_ftcontradiction_cosine.py

# 6. (Optional) Training intensity sweep (~3 hrs on CPU, 5 step counts × 5 seeds)
python3 -u exp_training_intensity.py

Results are saved to results/. The dataset is generated into data/ (gitignored).

Expected runtime on Apple M-series CPU: ~75 min per seed (main) + ~30 min (extended) + ~7.5h for 5-seed averaging.

---

Files

generate_rank_dataset.py   — generates the corpus and all 11 condition datasets

exp_svd_rank.py            — main experiment (run this first):
                             trains 11 models, verifies learning, selects layer,
                             computes SVD spectrum and Cf,k at layer 11

exp_svd_rank_ext.py        — artifact checks (run after main experiment):
                             answers three follow-up questions:
                             (1) Is style collapse a layer-11 artifact?
                                 → pairwise cosines at layers 3, 6, 11
                             (2) Does logit-weighted SVD improve Cf,k?
                                 → SVD on ΔA·W_U^T vs standard SVD
                             (3) Is the behavioral signal specific to layer 11?
                                 → Cf,k at layers 3, 6, 11

exp_baseline_alignment.py  — artifact check (4): baseline alignment test
                             does the dominant ΔA direction pre-exist in the
                             baseline model's activations (variance sink)?
                             → cosine of top-1 ΔA direction vs top-3 PCs of
                               A_base for C1-only, FT-Contradiction, FT-Random

exp_ftcontradiction_cosine.py — direct cosine between FT-Contradiction and
                             each C1–C5 top-1 ΔA direction (5 seeds)
                             → confirms collapse extends to semantically
                               opposite fine-tuning (cosines 0.971–0.982)

exp_training_intensity.py  — controlled training intensity experiment:
                             C1-only at steps {100,250,500,1000,2000}, 5 seeds
                             → s1_frac, Cf,k, and KL as functions of training duration

results/
  exp_svd_rank.json             — full results from main experiment
  exp_svd_rank_ext.json         — full results from artifact checks (1–3)
  exp_baseline_alignment.json   — results from baseline alignment check (4)
  exp_ftcontradiction_cosine.json — FT-Contradiction direct cosine results
  exp_training_intensity.json   — training intensity sweep results
report/
  svd_rank_report.pdf      — 16-page report with all tables and interpretation
  svd_rank_report.tex      — LaTeX source

---

Experimental Setup

• Model: GPT-2 small (124M parameters, 12 layers, d=768)
• Baseline: Unmodified gpt2 checkpoint (no fine-tuning); all ΔA matrices are computed as fine-tuned minus baseline activations on the same 200 held-out probes
• Layer: Layer 11 (final transformer block), selected automatically as the layer with the highest mean ‖ΔA‖_F across conditions — it accumulates the most activation change
• Training: Full-parameter Adam, lr=2e-5, batch=4, 1000 steps, max 48 tokens
• Corpus: 1200 diverse English sentences across 32 domains
• Probes: 200 held-out sentences (not in training)
• SVD: Truncated, up to k=10 components on 200×768 delta-activation matrices
• Cf,k metric: Hook-based activation patching; KL divergence over full vocab

---

Limitations

• GPT-2 small only — findings may differ for larger or instruction-tuned models
• Synthetic stylistic concepts, not real domain fine-tuning (medical, code, etc.)
• Five seeds (42–46) — Cf,k std ≤ 0.012 for concept conditions, 0.022 for FT-Random; eff-rank constant across all seeds
• Mean-pooled activations — position-specific effects are averaged out

---

License

MIT