If those who have gone astray will not harm the guided (Qur'an 5:105), why were many Prophets harmed?

> O you who have believed, upon you is [responsibility for] yourselves. **Those who have gone astray will not harm you when you have been guided.** To Allah is your return all together; then He will inform you of what you used to do. -- Qur'an 5:105 I feel I'm misunderstanding the sentence in bold. Several Islamic Prophets seem to have been harmed by those who have gone astray; as a few examples: - Prophet Yusuf was thrown down a well, was sold as a slave, and was imprisoned for many years. - Prophet Nuh 's and Prophet Lut 's wives "...were under two of Our righteous servants but betrayed them" (Qur'an 66:10 ). - Prophet Yunus was thrown off a ship, to be consumed by a fish (Qur'an 37:142 ). Presumably, it's safe to consider that these Prophets were guided, yet harm came to them. **Question**: If those who have gone astray will not harm the guided, why were many Prophets harmed?