Why Buddha was not worried about his family after enlightenment?

Of course, after getting enlightened, One is free from worry: now the person is in higher dimension and is always happy, as he knows how perfect everything is. But still Buddha's family was there; I mean, wasn't his family his responsibility? I remember when the Buddha came back to the palace and met his wife: she asked, "just tell me, if it was possible to get enlightenment in the palace." How necessary is it to leave our families to practice, and if it is not necessary why didn't Buddha just come back? I always feel sad when I think about Siddhārtha Gautama's wife Yashodhara. If one is enlightened, he can not hurt anyone feelings: but Buddha hurt Yashodhara's feelings? I know I am incorrect somewhere, because after all He was enlightened, so he can not take wrong decision.