What purpose does a normalizer serve in a loss function?
Going through some TensorFlow tutorials off of CognativeClass.ai, and in one of their linear regression examples, my linear regression wasn't working until I put a normalizer into the loss function.
Code here: Link
See the 'nf' variable on line 28. What does it do in line 30? It seems like adding 'nf' to both sides of the equation would accomplish nothing, but without it my results are hot garbage.
Posted Dec 18, 2017, 10:20 PM UTC
12
I think it’s just MSE, with both sides multiplied by nf. What you get is $$(nf * a - nf * b)^2$$. You can rewrite that as $$(nf * (a - b))^2$$, and subsequently $$nf^2 * (a - b)^2$$. It divides your loss by $$1^-2$$. This acts like a ‘learning rate’ but you should adjust the real learning rate (first parameter given to the optimizer) instead of adding this multiplier. I think if you divide your current learning rate by 100, you won’t need the multiplier anymore.
Thanks, I tested it with nf = 1 and the optimizer at 0.0001 instead of 0.01 and you're right, it generated the same results.
Not sure why they did that instead.