Recently built my first LLM and im wondering why there hasn't been more innovation on moving away from transformers and gradient descent?

So please excuse my lack of knowledge in this area as im new to AI/LLMs but I just recently build my first micro llm and I dunno something about them seems wrong.

Is the industry stuck on transformers and gradient descent because coming up with alternatives is a hugely difficult problem or is the industry just having blinders on?

I like a lot of the research about sparse models that use hebbian/oja and i know these come with challenges like catastrophic interference. But this seems like a very solvable problem.

Anyways im starting to tinker with my micro llm to see if I can get rid of gradient descent and traditional transformers and see if I cant make a sparse model based on hebbian/oja at the very least in a small scale

Again pardon my nativity, my expertise is mostly in backend systems and architecture. I have very little exposure to AI/LLMs until recently.