Cancelled a sit for the first time and feeling guilty

I canceled a sit that was meant to start in two days. The owner booked me for house sitting and didn’t tell me until the meet and greet that he would prefer I don’t spend the night. The sit required me to care for the dog from 6 am - 11 pm daily and was 25-30 mins from my house (depending on traffic). I told him I’d prefer to stay over due to the commute and he said that would be fine but that he would lock his bedroom door (?) which I responded would be no problem and he could lock all the doors except for one bathroom if that made him more comfortable. He also told me he would create a code for me on the security system because he didn’t want to give me the general one (I’m a 5’3 girl with 50+ reviews and only 5 star ratings, I don’t know what about me made him so uneasy 😭). That was about two weeks ago. Today, two days before the sit, he messaged me to tell me his wife wasn’t comfortable with me staying overnight. I don’t know, if I’m going to be in your home for 17 hours a day, I don’t see the issue in me staying an extra 7 hours with my eyes closed. Unfortunately after doing the math on both my time and money spent on gas / earned on the sit it would have been more trouble than it was worth so I had to cancel. I’ve never cancelled a sit before and I feel horrible, but that was too much for me