Is removing the whole gallbladder worth it for just 1 stone?

I had my first attack last month and I didn't believe it was a gallstone issue despite what my dr believed so i kept eating as normal and had no issues untill this week. Just for reference, I had steak, fast food, cheesy pasta, all with no issues over the corse of a month. And then Thursday I had a sandwich and for 2 days I had dull pain in the gallbladder region. Sunday I stupidly had a mug of hot chocolate and was in intense pain for about 18 hours. I reached out to my doctor and they had me do labwork to check my liver, which was fine but he wants to refer me for surgery and said it needs to come out. Am I crazy for not wanting it removed after just 2 attacks, or is it that the 2 attacks were so close together that I should be concerned?