What's better? using +4 Physical Attack up or +4 Elemental attack up? (plus little story question)

So I was playing with a bud in Deep of Night (both of us Ironeyes) and even though I was using elemental attacks and Evergaol Invader relics and he was playing only +4 physical attacks. His stats showed me that his damage was better than mine (even though I was using the bosses weakness and completed 4 Gaols and 3 Invaders).
Anyone able to explain this? His better damage has been messing me up lately