Reconciling metta & avoiding fools/immoral people

The Buddha repeatedly said to avoid ignorant, immoral people as much as possible, but isn't this - at least a little - opposed to kindness? I associate kindness with being benevolent, caring & somewhat open. Am I missing here something? Thanks