What is the basis of my fractured understanding of "Sola Scriptura"?

As a Catholic, I'm often baffled by other Christians who base their faith purely off personal interpretation of the Bible as the be-all-end-all proof for everything under the sun (which is what I mean by Sola Scriptura). I understand the need to figure out what's going on in the bible and the need to live your life according to wisdom of the prophets and saints and especially heed the words of Jesus. But where did the idea come from that everyone can interpret the bible for themselves? Is this just a common Catholic prejudice or a misunderstanding?