Why is there a lack of consistency when it comes to Genesis?

Whenever Genesis conflicts with natural science, I’m told it’s allegorical or poetic not meant as a literal account. But when it comes to things like human suffering, it shifts back to a literal story: Adam and Eve are described as real historical figures whose actual disobedience brought brokenness into what was once a physically perfect world.

Is there a way people reconcile these conflicts? Are there two main sides within Christianity on this, or is it usually the same people switching between metaphor and literal depending on what's convenient?