I wouldn't let me husband check my phone, and now I think my marriage is ruined

So my(32F) husband(31M) found a condom wrapper in our room. It was one he never would have used so he knew it wasn't his. He was suspicious and confronted me immediately. He wanted to check my phone to see if there was anything suspicious there.

As it turns out, my brother who is too cheap to buy an A/C unit for his room decided to bring a girl over and have sex with her on our bed where it is cool from the A/C we have. He left the wrapper hidden in one of my drawers which my husband found while looking for some floss.

I refused to let him see my phone. I told him it was a boundaries thing. He did not believe it was my brother partly because the day before I cancelled hanging out with him to hang out with a friend of mine, who is female which made the suspicion worse (what if it wasn't a female friend!?). It caused a huge fight, and now my marriage might be over.

We were already on thin ice, working through problems. We had been separated but hanging out 4-5 times a week and we had finally decided to cohabitate again starting September first. Seems like that is over :/

How do I fix this?