The Tower of Babel is a clear misunderstanding of how to reach God on the part of the city of man. They assumed that we could God physically, when in reality the means God has provided for reconciliation with Him is through humility. Not by raising ourselves up but by lowering ourselves to we come to see God.
When God "came down" to confound speech at Babel, this did not mean that He moved geographically, but rather that He stooped in dignity to move among men. This was part of the reason He was so angry at the idea of the tower, for He is everywhere and does not need to be built "up" to.
God speaks to His attending angels in a way different from that in which He speaks to us: namely, He doesn't have to dumb it down or filter in any way--He speaks and they obey.