Why was having concubines not a sin like adultery?

The Old Testament records many instances of men having concubines or many wives. Why did having concubines not fall under the sin of adultery? 2 Samuel 5:13 - "After he left Hebron, David took more concubines and wives in Jerusalem, and more sons and daughters were born to him." Genesis 25:6 - "But while he was still living, he gave gifts to the sons of his concubines and sent them away from his son Isaac to the land of the east." (Abraham) I suppose the issue here must've been consent between all parties.