Range needed to find low minimas are much higher than expected

Beginner question 👶

Hi! I started programming quite recently and one of the projects I made was a library for creating, utilizing and training neural networks.

However, I have come across a recurring issue; for the vast majority of problems I create networks for, I need to use a far greater range of randomization than expected.

To cite an extremely simple example, for an XOR type problem, giving a range of -1;1 (for initial randomization) doesn't allow the model to go under 0.5 loss (Cross-Entropy loss, so barely guessing) even after 200+ attempt on 10k epochs each. To get satisfactory results in a small amount of time (Loss < 0.05), I need to select a far greater range (ex: -10;10) which I find extremely odd.

I have checked numerous times in my randomization methods specifically but can't find any issue with it so I doubt the issue is there. And I mainly wanted to ask if there was a theoretical reason why this is happening.

And yes-, I did see the fact that the sub guidelines encourage to post the code, but frankly I don't think anyone wants to go trough 2000+ lines of code (last I count).

P.S: I'm not too sure under which flair this goes so I put it under beginner question, no idea if it's truly beginner or not, I don't have much experience.

Posted Jul 13, 2025, 5:07 AM UTC

32
Loading comments…