Modifiers question

Do the modifiers (+5 swing damage, -2 armor, etc) apply to the damage shown, or is the damage shown on a weapon the damage with the modifier already applied?

Edit:

Side-question, does anyone know if there's a damage penalty when wielding one/two-handed weapons in one hand and how harsh it is?