One particular problem is the accepted belief that is something is authentically Jewish, it can not be idolatry. And if some is not Jewish, then it automatically comes under the suspicion of being idolatrous.
But, in fact, an idol of a gentile can be nullified. That is if the gentile himself abandons its worship. Not Jewish idol.
The basic problems with idolatry are three things. One is the idol itself, another is things offered to the idol and the last is things that are vessels or ornaments made to serve the idol. People can be things offered to idols, and they do not have to agree to it. As in fact in the days when people were, in fact, offered to idols it was never the case that they agreed. Rather by the time they realized what was happening it was too late to turn around. By joining their cult, one becomes a thing offered to their idol.
And people can be idols themselves as we see in Sanhedrin pg 63 האומר עבדוני ואמר כן חייב. (A person says, "Serve me" is killed for seducing to idol worship. A person that agrees and says "yes" is also killed for worshiping an idol.) That is not only is the person that says "Serve me" is an idol and is killed for being a מסית ומדיח, but also the person that agrees and says, "yes" is also killed for serving an idol.
Idols an the vessels that serve them and food and vessels offered to them are forbidden to derive benefit from. They are also unclean [טמא]. What happens to the טומאה uncleanliness if the people that served it nullify it? In all the above cases the uncleanliness disappears except for food offered to the idol. That obviously remains forbidden, but the uncleanliness is a doubt if it goes off.
An idol that is worshiped by gentiles can be nullified and an idol that is worshiped by Jews can not. What happens to the טומאה uncleanliness? This all starts in Tractate Avoda Zara 52a. R. Yochanan asked R. Yanai what happens to the uncleanliness of food offered to an idol. The question is asked why did he ask about food? Why not ask about vessels? Vessels are not a question since they can be made pure by dipping them in a river or fresh spring thus the "Tumah" uncleanliness also goes off. Why then did he not ask about the idol itself? The idol itself is not question for since its status as an idol can go off of it when people no longer worship it, then its uncleanliness also goes off. But food is a doubt because it has no way of getting clean by dipping it in a river or ocean or spring.
There the Rambam and Tosphot and the Raavad all hold vessels that were offered to the idol and vessels used to serve the idol can have their uncleanliness taken off. But Rashi explains in that Gemara that vessels can be made pure just like their use for idolatry can be nullified. So Rashi obviously is explains our gemara here as referring on to vessels that are used to serve the idol, not to vessels that were offered to the idol since their prohibition for use never comes off.
Thus it is important not to serve a Jewish idol since the uncleanliness and the prohibition can never come off. We also see this in Sanhedrin circa 65. When a Jew leaves serving a Jewish idol he dies (because his source of life is cut off.) We actually since this in gentiles also. Even a gentile when he leaves a cult that he was involved with, he looses his life source and dies spiritually--and sometimes physically. This you always see in people that break away from cults. They never get back on track no matter how hard they try. Or they just go and join some worse cult.