Can't Increase Music Volume with Siri on Watch

When I'm riding my bike, I use my phone to play music via a Bluetooth speaker. I can use my watch and Siri to pause and start the music again, but I cant get Siri to increase/decrease the volume. Siri will confirm on the watch that it's been done, but the volume doesn't change. Does anyone know if it's possible to use Siri on the watch to change the Bluetooth/phone volume?