Stopping a music app

I've always had this problem with my phone but it's even harder to deal with on my watch. Mainly, when I'm done listening to something on a music app, how do I get the app to basically "exit" and not play anything until I initiate it again? Right now I can pause the app and swipe it away, but sometimes if I put some earbuds in or do something else on the watch, it will start playing again from where it left off. On my phone I often do a Force Stop to tell it to just go away, but that s really clumsy to do on my PW4. Any ideas or tips on this behavior?