Why not make up your own times to pray from non Muslim

When a non Muslim learns that Muslims are required to pray at particular times of the day, his reply was you should be able to decide for yourself when to pray. For example, Perhaps you are busy today or tired so you pray less and pray more on other days. Basically you choose how many times to pray per day and when. Why is this not a good idea from an Islamic perspective?