How much does publisher relevance matter when evaluating a link?

I've been putting more weight on the relevance of the site and the actual audience behind it when evaluating potential links.

A site can have strong metrics but if the topic is only loosely related and the audience has little overlap, I am not always convinced the link is particularly valuable.

At the same time, I wouldn't want to dismiss a relevant publisher just because its traditional authority metrics aren't as impressive.

How do you consider publisher relevance, organic traffic and authority in your own head when determining whether a link is worth pursuing?