I started it and now I have stopped

A little while ago I said to my partner that I wanted to bring in another girl to have some fun with us I explained it was a one time thing in a place where we would never see this person again he initially said it might not be a good idea but after talking about it we agreed to go through with it so I found the person to do it with but when the day came the person pulled out for me it was a big let down I had built up all this excitement to have it fall apart and have since not been interested in trying it again but the whole thing has sparked something in my partner and he seems interested in doing it but I don't want to and I don't know what to do