The dwarf giant theory?

I have a theory that dwarves used to be giants.

All of their oldest structures are ridiculously tall and vast.

They also have a build similar to giants, just smaller.

I belive that over the millennia of mining, they got smaller as to adapt. Give me your thoughts on this theory.