4 nights in Mallorca before 3 nights in Madrid. Where to stay?

Trying to decide if we should stay in Soller or Palma while visting Mallorca. We like a mix of lounging, convenience to good coffeeshops and restaurants, and some activities. We'll be renting a car, so don't need to be super close to a beach. We're moreso looking for nice sights, but still not too disconnected from things to do.