Bath in May Plus Where Else?

A friend and I are headed to Bath for the Badminton Horse Trials in May. We know we'd like a couple of days to explore Bath in detail before the horse trials while we recover from jet lag (coming from the west coast of the US). Beyond that, we're having a hard time narrowing down what / where else we should include. We'll have up to a total of 2 weeks, roughly May 3-15.

Things we're interested in: gardens, estates / castles, ruins, pretty towns to walk in, museums (including quirky ones), good food, and pretty villages. I'd also love to get some time in a kayak and/or a bike tour. We want to be careful we don't over schedule ourselves. We are planning to rent a car for at least part of our time.

Ideas? Recommendations? Can't miss places? Great places to stay? Pubs / markets / restaurants we can't miss? Mostly we need some help getting started so that we can start narrowing this down and get some lodging reservations made beyond our nights in Bath.