How many Wisemen were there?

The wise men are commonly thought of as three men, each bearing a gift for the baby Jesus. When in fact the scriptures don't indicate exactly how many wise men there were, only what gifts they brought. There could have be four, five, or more wise men that each brought either gold, frankincense, or myrrh. Or, there may have only been two wise men, and both of them could have given the Christ child three gifts. > "¶And when they were come into the house, they saw the young child > with Mary his mother, and fell down, and worshipped him: and when they > had opened their treasures, they presented unto him gifts; gold, and > frankincense, and myrrh." (Matt 2:11 ) Is there any indication that there were more or less than exactly three wise men? Or has it always just been assumed that the fact they brought three gifts indicates there were three men? If so, when did this association begin?