I brought it up though I wasn't that angry today when I brought it up. We talked about emotions and my fear of being manipulated and trust issues.

If he is any good, he should be able to handle it.

But what you should do is bring it up as something that you are experiencing. Hopefully, you will both explore the reasons, and any subconscious factors causing this. This anger is of significance in the therapeutic process.

You really should bring it up and try to resolve it with your therapist. I am pretty sure he would want you to bring it up. Just do it with courtesy.

I'm really angry at my therapist thinking about him at the moment, and I have a desire to express my anger towards him when I see him next, though I think my anger is also unfair and I am scared to get mad. And I worry, 'what if he kicks me out of therapy.' 

Anger at my therapist!