Why God allowed some captives from Judah to be exposed in idolatry acts of Babylon

In the book of Jeremiah, God wants to punish Judah for different acts of sin like idolatry. God *will* use Babylon as the instrument to fulfill those prophecy and to stop them in worshipping their false gods. But in Chapter 51 of Jeremiah, God will also punish the Babylon because of doing like what Judah did by worshipping idols. Then why God allowed some of the captives/remnants from Judah to be at Babylon *(which is stated to be worshipping idols)* if God wants them to stop worshipping other idols or false gods?