When does an unarmed build come online?

I’m new to the game and wanted to switch to unarmed at level 6 when i got the iron fist perk. However, when i took it and switched the dropoff from a fine sword was astronomical. Is there some point where being a punch machine becomes viable?