Could you save energy in large coolers by pulling colder air from outside?

I was wondering if there is any significant fundamental engineering theory that would prevent larger coolers (such as cooled warehouses) from being more efficient if they pulled cold air from outside during winter. What I mean by this is that, if you lived in a cold Northern climate, could you just suck the cold air from outside the building to boost the efficiency of the cooling systems. I know some about refrigeration systems, and I know they run off of temperature differences between input and output (larger the difference in temps. the greater the efficiency), but I want to know if I'm missing something.