Crack that whip

I was recently asked in a small group of white people being the only black person, if my manager was cracking the whip on me. My manager is also white, I felt so uncomfortable and laughed it off. Should I just let it go or say something?