Hypothesis: Low-Rank Adaptation (LoRA) enforces highly correlated, low-dimensional rotations of ReLU gate hyperplanes (normals), rather than allowing the independent, local fragmentation that Full Fine-Tuning produces.
This repository contains a rigorous PyTorch experiment that studies the geometry of ReLU neural network adaptation. The core question is:
When adapting a pretrained MLP to a new task, does the rank constraint in LoRA fundamentally change how the decision-boundary geometry evolves, compared to unconstrained Full Fine-Tuning?
We answer this through a controlled topological experiment: a "Bubble Flip" (XOR patch) is surgically placed inside a region of constant ReLU gate activation, empirically demonstrating that the output head alone cannot solve the task — hyperplane movement in fc1 is mandatory.
Scope note: This experiment demonstrates empirical geometric effects of low-rank adaptation under a controlled synthetic setup. It does not constitute a formal proof about all neural networks.
This repository provides geometric evidence supporting the claim: Low-rank adaptation restricts the dimension of the hyperplane deformation field.
This is stronger and more mathematically precise than the typical statement that "LoRA regularizes fine-tuning" (which is vague). Low-dimensional deformation is measurable, as shown in this repo.
Instead, it evaluates how the decision partition deforms during adaptation. Low-rank updates restrict the directions in which hyperplane normals can move. This induces a correlated deformation of the partition, rather than the chaotic, independent motion of individual hyperplanes seen in Full Fine-Tuning.
Input x ∈ ℝ¹⁰ (data lives on a 2D plane embedded via fixed E : ℝ¹⁰ → ℝ²)
│
▼
fc1: Linear(10 → 32) ← Normals (rows of W₁) define ReLU hyperplanes
│
ReLU
│
▼
fc2: Linear(32 → 1) ← Binary classification head
│
▼
ŷ ∈ {0, 1}
- Embedding: Input
x = E @ u, whereu ∈ ℝ²is the true 2D coordinate.E ∈ ℝ^(10×2)is a fixed orthonormal basis (QR-factored random matrix). - Activation:
nn.ReLU()(strict; no leaky/ELU variants).
| Task | Description |
|---|---|
| Task 0 (Pretraining) | Binary circle: ‖u‖ < 0.60 → class 1 |
| Task 1 (Adaptation) | Circle plus a "Bubble Flip" XOR patch of radius 0.08 placed at a constant-gate region inside the circle |
The XOR patch flips labels for points within radius rb=0.08 of the bubble center u_c.
-
Topological Trap — After pretraining,
find_bubble_center()searches for a pointu_cwhere all 32 ReLU gates offc1remain constant throughout the entire bubble. This guarantees the geometry is locally flat. -
Head-Only Assert — The base model (no training) is evaluated on the bubble region under Task 1. Since the gate pattern is constant and the label is flipped, accuracy = 0.0. This demonstrates that fc2 alone cannot solve it under this configuration.
-
Frozen Output & Bias — During Full FT and LoRA:
fc2.weight,fc2.bias, andfc1.biasare all frozen. Only the normal vectors (columns offc1.weight) are allowed to change. -
Target-Loss Matching — Both methods train until reaching the same target loss (
0.15) rather than a fixed epoch count — ensuring no unfair comparison of underfitting vs. convergence. -
LoRA on fc1 only —
W_eff = W₀ + (α/r) · B @ A, whereA ∈ ℝ^(r×d_in),B ∈ ℝ^(m×r),r=2. The update is provably rank-2.
| Level | Metric | What it measures |
|---|---|---|
| 1 | Update-Matrix-Rank | rank(ΔW) and stable_rank(ΔW) in weight space |
| 2 | Hyperplane-Rotation-Rank | stable_rank(ΔN) where N = unit-normalized rows of W₁ |
| 3 | Region Creation vs Movement | Differential count of unique gate patterns on a 2D grid before and after adaptation |
| 4 | Gate-Drift / Adjacency | % of area changed; Jaccard distance of the Polytope Adjacency Graph before/after adaptation |
| 5 | Discrete Boundary Curvature | Mean, median, and 90th percentile (p90) turning angles of the boundary's geometric normal vector. |
| 6 | Line-Crossing Complexity | Crofton-proxy: avg. sign changes of prediction across 256 random lines |
The mean curvature of the decision boundary is nearly identical between LoRA and full fine-tuning. The difference appears primarily in the tail of the distribution. Full fine-tuning produces slightly more extreme turning angles, while LoRA suppresses the largest boundary kinks. This supports the interpretation that low-rank adaptation limits extreme local deformations while preserving global flexibility.
The rotation-rank metric is essentially measuring the dimension of the normal fan deformation of the ReLU partition. In terms of tropical and polyhedral geometry:
rank(ΔW) → dimension of Newton polytope deformation.
LoRA's low-rank structure geometrically bounds the complexity of the topological changes applied to the data manifold.
# Create conda environment
conda create -n plora python=3.12 -y
conda activate plora
# Install dependencies
pip install -r requirements.txt
# Run all experimental tracks
make all
# Or run them individually
make baseline # Core baseline experiment
make bias # Bias frozen vs trainable ablation
make depth # Depth composition study (with geometry metrics)
make rank_sweep # LoRA rank sweep r = [1, 2, 4, 8]
make random_control # Random low-rank update baseline
make input_dim # Ambient dimension ablation (d_in = 2 vs 10)Outputs:
results/logs/— machine-readable numeric results (JSON)results/figures/— generated plots
| Metric | Full FT | LoRA (r=2) |
|---|---|---|
rank(ΔW) |
10 | 2 |
stable_rank(ΔW) |
1.11 | 1.00 |
| Rotation rank (10D) | 10 | 10 |
| Rotation stable-rank (10D) | 1.23 | 1.25 |
| Gate-Drift | 62.3% | 56.0% |
| Adjacency Graph Drift | 63.9% | 50.8% |
| Boundary Curvature (mean) | 2.03 rad | 1.98 rad |
| Line-Crossings/Line | 1.18 | 1.13 |
| Gate Regions (Created) | 56 | 37 |
| Gate Regions (Retained) | 103 | 117 |
| Task-1 Global Acc | 0.970 | 0.965 |
| Task-1 Bubble Acc | 0.937 | 0.932 |
See evaluation.md for full analysis and figures.
To strengthen the empirical claims and rule out alternative explanations, the repository includes four control and ablation experiments:
Tests LoRA ranks r = [1, 2, 4, 8]. For each rank, measures stable_rank(ΔW), gate_drift, adjacency_drift, line_crossing, curvature, and region_count. This lets us determine whether geometric effects scale with rank or are binary (any rank vs full FT).
What it can show: Dose-response curve between LoRA rank and geometric boundary complexity. What it cannot show: Whether the same relationship holds for all network architectures or datasets.
Compares Full FT, LoRA (trained), and a random low-rank update ΔW = scale × (B @ A) with random B, A matched to the LoRA update norm. The random control is not trained on Task 1.
What it can show: Whether LoRA's geometric effects require learned structure, or arise from rank constraint alone. A random rank-2 update will have matching rank but different geometry. What it cannot show: Whether random low-rank updates could also solve the task (they won't, since they are untrained).
Extends the depth study (depths 1, 4, 8) to report full geometric metrics — curvature, line_crossing, region_count, gate_drift — in addition to rank and accuracy. Tests whether the low-rank geometric constraint propagates through deep nonlinear composition.
What it can show: Whether the boundary complexity remains correlated at deeper architectures. What it cannot show: Behaviour beyond the first-layer LoRA configuration tested here.
Compares d_in = 2 (data lives directly in 2D) vs d_in = 10 (standard 10D ambient embedding). The data manifold (2D circle + bubble) is identical; only the ambient parameter space changes. Tests whether rotation_rank, curvature, and region creation depend on ambient dimension.
What it can show: Whether the observed effects are a property of the 2D data manifold or the 10D ambient space. What it cannot show: The effect of ambient dimension in high-dimensional real-world tasks.
One conceptual limitation of the 1-layer architecture is that it isolates geometry nicely, but leaves open the question: Does the effect survive deep composition? Deep ReLU networks can create exponentially many regions.
To address this, the codebase includes experiments/run_depth.py, where LoRA is applied only to the first layer across depth = 1, 4, and 8 networks to test whether the geometric restriction imposed at the first layer survives deep composition.
Empirically we observe that even at depth 8:
- LoRA updates remain strictly low-rank
- The model still solves the task (>96% accuracy)
- Boundary deformation remains correlated rather than chaotic
This suggests that deep nonlinear composition can amplify a low-dimensional deformation field without destroying its structure.
.
├── models/
│ ├── mlp.py # Base MLP model
│ ├── lora_layer.py # LoRA implementation for first layer
│ ├── deep_mlp.py # Deep MLP model and LoRA variant
│ └── utils.py # Data sampling and training utilities
├── metrics/
│ ├── gate_drift.py # ReLU gate pattern drift metric
│ ├── line_crossing.py # Crofton complexity proxy
│ ├── rotation_rank.py # Update rank and rotation metrics
│ ├── curvature.py # Discrete boundary curvature
│ ├── adjacency.py # Polytope adjacency graph drift
│ ├── region_count.py # Region creation vs movement (tuple hashing)
│ ├── normal_motion.py # Normal-vector movement rank
│ └── random_low_rank.py # Random low-rank perturbation baseline
├── experiments/
│ ├── run_baseline.py # Core baseline experiment
│ ├── run_bias_ablation.py # Bias frozen vs trainable ablation
│ ├── run_depth.py # Depth composition study (+ geometry metrics)
│ ├── run_rank_sweep.py # LoRA rank sweep r = [1, 2, 4, 8]
│ ├── run_random_lowrank_control.py # Random low-rank control baseline
│ └── run_input_dim_ablation.py # Ambient dimension ablation d_in = 2 vs 10
├── scripts/
│ └── plot_results.py # Plotting suite (baseline, rank sweep, depth, random control)
├── results/
│ ├── logs/ # JSON results and logs
│ └── figures/ # Generated plots
├── requirements.txt # Python dependencies
├── Makefile # Build targets for all experiments
├── evaluation.md # Scientific report & interpretation
└── README.md # This file
Experiment design inspired by:
- Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (2021)
- Montufar et al., "On the Number of Linear Regions of Deep Neural Networks" (2014)
- Raghu et al., "On the Expressive Power of Deep Neural Networks" (2017)