Are there any references that people of the book have actually had ties to the Ka'ba?

We know that Ibrahim rebuilt/built the Ka'ba. And it was a place of worship for the prophets after him. Doesn't this mean Ka'ba was a place of worship for the Christians and Jews? That is should be revered among them as well? - If so then are there any references in their books regarding Ka'ba/Mecca being a holy place? - Has there been any Islamic researches regarding this? Are there any narrations about how the prophets of their time visited the Ka'ba?