Buddhism is not fatalism?

According to the Buddhism's theory, one man's fate is destined by the past's karma. Suppose that he faced an unfortunate incident at birth, which is due to past karma. Besides, his genes are inherited from his parents. In other words, his brain is also initiated by past karma. In this condition, all his ideas are also affected by the experience he has had, which is determined by past karma. To elaborate more, some are clever, some less, which can again be explained by karma. But if the conventional human brain can be so strongly determined by the existence of karma, how can you say buddhism is not fatalism?