My first attempt at building an ML optimizer from scratch in C++ (Newton's method & Lagrangian)

Hi everyone!

I've been diving deep into the "Deep Learning" book by Ian Goodfellow, Yoshua Bengio, and Aaron Courville and i decided to implement the core ML logic in C++ from total scratch. No external dependencies, just raw math.

I building my own RawMath class. I implemented sigmoid and exp() using Taylor series just to keep it zero-dependency.

For the optimization, i didn't want to stop at basic SGD. I implemented Newton's method using second-order derivatives because i wanted faster convergence. I added Lagrangian constraints to handle boundaries and step adjustments and a velocity/inertia component to the weight updates to maintain stability during gradient shifts.

This is just the start. I’m planning to implement PCA (for image compression) and Linear Regression in the same "zero-dependency" style.

How can I make this even faster? Are there obvious C++ optimizations I’m missing for this specific Newton-Lagrange loop?

GitHub Link in the comments