Idolatry within Islam

If Islam strives against idolatry, can you explain why when Muhammad returned to Mecca and destroyed the false idols, he intentionally left the Ka'bah which is now centric to Islam. I think I understand that the Ka'bah seems like more of a connection to Allah though Abraham, but praying in the direction of the Ka'bah, isn't that somewhat hypocritical?