Bardric question (trying to avoid spoilers but also contains a spoiler)

>!A while ago I saw someone say that if you don't take Bardric on a certain quest with you, he attacks you. I wasn't romancing him at the time but now I am and want to avoid this. I don't want to use him as my ally on every choice as I like to mix it up. Can someone tell me which chapter/scene I need to bring him with me? !<