Day trip to Kamikochi and Hirayu no Mori

We're going to be in Takayama for a few days in mid-May and wanted an outdoor onsen experience. We then decided to turn it into a day of a little exercise and a little relaxation so we thought we'd spend the morning walking in Kamikochi. I found some information online that made it sound like doing both would be impossible, but I've looked at bus times and it seems like it would work just fine, so I figured I'd ask here.

It looks like the first bus leaves Takayama at 6 and after a transfer at Hirayu Onsen, at around 7, would get us to Kamikochi by 7:30. The last bus doesn't leave Hirayu Onsen for Takayama until 19:23. We're not worried about the early time since we plan to be up early and to bed early anyways. If we went to Kamikochi early, hiked for 4-5 hours, headed back to Hirayu no Mori and spent 4-5 hours at the onsen, that seems like enough time to enjoy both. I've never been to either though, or any onsen for that matter, so it is possible I am missing something?