Method to evaluate pop up message

Hi there,

In our team we are currently discussing to include a pop up message that is shown to the user if they want to cancel their subscription, basically reminding them what benefits they have with the subscription and trying to convince them to stay.

Now, we would like to test the pop up first with users to see if they are disturbed by the pop up and if the message would actually convince them to stay. Since here, we are evaluating their subjective attitude towards the pop up I assumed we need quantitative data to make conclusive statements (i.e. unmoderated quantitative usability testing or survey). However I am not sure if a qualitative usability testing would be more appropriate to capture feedback and reaction of the users to the pop up. What would your advice be?

Thanks a lot!