How can I practise listening?

I find myself building mental associations while someone is speaking to me. I might ask them a question and while they are talking, I’ve already left to form some mental map (prapanca I suppose) i.e. I don’t listen much (I think I am more visually-oriented). Is this really bad or just my imagination? It does appeared socially impolite and I do it often, so maybe there is a functional reason (like too much information to process). It would be rude to interrupt or not look and pay attention. So, the conversation might end up awkward or repetitive. Perhaps, it is a sign I don't really care about their response which is terrible especially since I asked the question. But the conversation can be lost in a split-second as a result of a visual que, a background sound or any distraction. What do you think? How can I progress to be a better listener?