Dispassion in Buddhism

Is something along those lines somewhere to be found in? I am aware of the drawbacks mentioned in MN.19 , but could this way of thinking, that is, the shortness of pleasure not lead to isolation/life rejection? The Buddha frequently used the word dispassion towards the world, but I HIGHLY DOUBT HE HAD AVERSION to worldly things. This is a common phenomenom that many monks also display ,that is, that they feel aversion to the world. How does one deal with such a (mind) problem to not to drift into negativity to wordly affairs?