Frieza was not his hardest fight.

His 4 hardest fights were in order: Radditz, he died to fight his brother he didn't remember. Vegeta: he was injured so bad and it took so long to gain the senzu bean he needed to recover. Cell: fought so hard Cell blue him up, killed King Kai, destroyed the dwarf planet, killed Gary and Bubbles. And King Kai and friends never got revived sad. Hit: he went blue plus Kaoken x10 which hurt his recovery and almost lost control of his abilities!

He could have died against Frieza but since he was the strongest Saiyan Frieza ever met, Frieza enjoyed torturing him!