Why not caulk around the electrical boxes?

This may be a stupid question, but say when you install a new outlet box in a finished wall, the gap around the box would it be acceptable to just caulk around it? saves time from mudding, sanding painting before putting cover on for tighter air sealing, or do you guys just leave the gap?