Shouldn't we care only "What I think" instead of "What I + Others think"?

Our own thoughts make something wholesome or unwholesome. Not what other's think. So, is there a need to ***care*** about what others think regarding a certain thing? We can do good for others, but they might think that we do something bad for them. Isn't it their responsibility to think in a wholesome manner? The title of this question should be "Should we care about what others think?", but I found a related question here: However, it doesn't have any answers regarding this case (karma). Kind regards