How bad is driving on a dirt road, really?

I had something of a debate with a friend- We disagree on the topic of how bad dirt roads are for cars, but neither of us are really qualified to make a judgement; we're both nerds in different fields.

So the prompt is this: Given a regular 4-door sedan (2005ish Honda Accord or Toyota Camry or Volkswagen Jetta), how much would you have to drive that car (going less than 50 mph) to see an appreciable amount of wear/tear on the car? How much would you have to drive on a dirt road to notably reduce the lifetime of the vehicle?

By "dirt road" I mean a well-manicured, level, brown dirt road, like you'd see off the side of a highway in Iowa. The specific roads of the conversation were roads that service farms near highways in Iowa. "Dirt road" could also mean a well-manicured, level gravel road.

Edit: A dirt road like this one.

I am under the impression that such dirt roads, at less than 50 mph, are within the "normal operating envelope" of the vehicle, meaning that the shocks, undercarriage, wheel wells, etc. would not see a notable reduction in lifetime from the occasional drive on such a road. My friend is under the impression that 20 minutes of driving would be very bad for a car.

So, who is right? How bad are such dirt roads for a typical 4-door city sedan? Should I really avoid driving on these roads at all costs? Could I drive an hour a day without seeing an appreciable effect on the lifetime of my car?