Am I following Buddhism as a sort of Escapism?

Some body close to me said to me, "you failed in the real world and so now you are chasing Nirvana and running away". It hurt me a little but set me thinking if he is right? The story is, I come from a Hindu background and long story short I got into terrible life situation. By some miracle I got stumbled on Ajahn Brahm's talk on youtube four years back. I started digging deep and now calling myself a Buddhist, I sitting on my cushion aiming Jhanas and Nirvana. But could he be right, imparting status quo worldly wisdom of 'I should fight, I should work harder, I should not give up on my worldly goals' I have given up my job and no relationship. I spend all my time meditating and listening and reading Buddhism. But have I taken this path as sort of run away from responsibilities. Did I even had a choice to begin with in doing what I am doing. Have I chosen this path or I am just blindly riding what has came to my way with no say of mine.