Hesitant to get into downhill

I grew up riding bikes all the time. Dirt jumping down by the creek, BMX, mountain biking, a bit of downhill in my late teens. I am very comfortable on a bike and in decent shape but I'm in my early 40s now.

I got the urge to downhill again and went up to Northstar and rented a bike and gear. It was sooooo much fun. I am really tempted to try and get back into it but man was it rough on the body. I mean part of it is just not being conditioned and also being at altitude, but it is absolutely a brutal sport. That day at Northstar I witnessed a broken arm and a broken femur.

I've never been super risk adverse in my activities but I always take the appropriate precautions with action sports like skateboarding, snowboarding, etc. Riding within my limits, not pushing things too hard, being mindful of when I'm tired, and learning how to fall (thank you skateboarding). I've had a good track record of avoiding serious injury up to this point. I know a major fall can be extremely challenging though - surgeries, physical therapy, etc.

If I'm gonna do it of course I want to have all the best protective equipment, helmet, etc. I'd likely just rent a bike a few more times unless I find myself going a lot more frequently. I think for some, maybe the risk is part of the thrill and enjoyment. That's not really what makes me love it though. Am I crazy for wanting to get into this sport??