Why are the driver where they are in cars?

Whether left side (driving on the right) or right side (driving on the left) the drivers are always on the 'inside' when they pass each other on a 2-way road. Isn't this more unsafe than if drivers were on the opposite sides?

Edit: I meant drivers, not driver