Campgrounds for Primitive-ish Camping in Colorado

My dad and brother want to go on a boys camping trip to Colorado. We want to stay in the Crested Butte and Gunnison area, preferably somewhere primitive and overland that we can still drive up to with a jeep, and near a river for fly fishing. In addition we’d also prefer somewhere that’s relatively isolated; we just don’t want to have to see or hear any other campers. Anyone know any campgrounds or parks that fit these circumstances.