Hitting opponent with racket?

I was reffing at a tournament this weekend, and was confused about a certain rule. Say I follow through and hit the opponent with my racket? I know it should be me who calls the let, but if I don't, how does play continue? Can the opponent call a let if this slows him down when he goes to retrieve your shot?

Thank!