LMC in Transformers
A fun experiment exploring whether Linear Mode Connectivity (LMC) exists in transformers.
Despite the highly complex loss landscapes that often occur in neural network training, there have been interesting phenomena and patterns shown empirically. Linear Mode Connectivity (LMC) is one of those phenomena—the observation that different solutions reached by Stochastic Gradient Descent can be connected with a linear path.
In this work, I study whether LMC occurs in Transformers. I provide empirical evidence across decoder-only transformers with varying sizes and model configurations, following GPT-2 (Radford et al., 2019; Brown et al., 2020).