Bike carry of X100v with 3 point harness, but in a bag?

Thinking of getting an X100v (maybe X100vi), I'd like to be able to carry it on my back on my road bike for my rides using a 3 point harness like the PS Rider Strap, but in a small waterproof/sweatproof bag/case, rather than exposed unlike most illustrations of this usage - is that likely to be possible and, if so, what bag/case would be good for this (needs to also fit the camera having a filter adaptor on it)? Although the X100v can be made weather-sealed, I'd still worry about sweat damaging the camera and smearing the screen/viewfinder or unexpected heavy rainfall (I assume would be okay in light rain?). Also a case would provide at least a minimal amount of protection in case of accidents.

The camera is not for taking shots whilst actually riding, but for when I've stopped to photograph something, though I still want pretty quick access to the camera. I don't want to carry the camera on the bike itself (too much vibration and I prefer not to use bike bags on my road bike), nor do I want to have to wear a backpack or hip pack.