How do you know what the appropriate greeting/leaving should be with another dude?

Maybe I’m overthinking this, but how do guys decide on the correct greeting? Handshakes for bosses and professional settings seem obvious, but once you get into friends of varying degree, brothers, cousins, dads, and in laws, there is something that feels right with each.

Some people get a nod, some a fist bump, some a handshake, and some a hug. Is this all just a split second vibe check? And what do you do when one person goes for a different greeting than the other, do you laugh it off or ignore the awkward and move on.

Throw in cultural norms and intercultural interactions. Is It better or worse if you try to mirror a cultural norm and come across as goofy?