Can a Muslim man take a concubine who's neither a Christian nor a Jew?

It's well known that, according to the orthodox Islam, a Muslim man can marry only with a Muslim woman, or a Christian woman, or a Jewish woman. Does this restriction apply to concubines? For example, can a Muslim man take a concubine who's an atheist, or a polytheist, or an agnostic etc. ?