New PyTorch optimizer for continual learning
This optimizer wrapper for continual learning is guided by the condition number (κ) of model tensors. It identifies and updates only the least anisotropic parameters to preserve pre-trained knowledge and mitigate catastrophic forgetting due to a synergy of factors: their inherent numerical stability makes them less susceptible to training noise, and their less specialized nature allows for robust adaptation without overwriting critical, highly specific pre-training knowledge, thereby effectively mitigating catastrophic forgetting of foundational capabilities (see the link to the paper in the repository): https://github.com/oswaldoludwig/kappaTune
Posted Jul 6, 2025, 7:37 PM UTC
20