A good deal of the problems involved with worship of tzadikim involves the problem of delusion.
Idolatry also is a problem from the standpoint of the Torah. And especially since they are in the category of מסית ומדיח, people that entice others to worship their false gods]

That is we don't know whether that particular tzadik has real revelations of if his revelations are delusions. And on the same hand he might very well have delusions and yet be very charismatic.

And the emotional appeal might be great while at the same time have zero validity objectively.
 It is hard to separate these variables.
And when they are trying to make converts they don't say they worship the tzadik.
On the contrary they will emphatically deny it.

The secrets are  only for the initiated.

A good deal of the difficulties is because of numinous reality. And numinous reality is has potent emotional appeal and it sometimes is from the realm of holiness and sometimes from the realm of the tree of knowledge of good and evil and sometimes it is simply delusions. The trouble with pseudo tzadikim is that delusions are allergic. People pick them up from their delusional leader.

Where does this problem come from? To me it seems clear that the people that were able to see the problem decided instead to be silent or acquiesce. Rav Shach was the only one who saw clearly and he was ignored and still is. [Along with the Gra.] Now some people have taken the Gra seriously. That is the Zilverman's in the Old City. But they are a small  minority. Some people take Rav Shach seriously, but that is only in Ponovitch. Outside of these places I have never heard of anyone that considers worship of tzadikim to be  a problem.

And why is this that they were silent? It was because their expertise was not in Jewish philosophy. People like Reb Shmuel Berenbaum thought of themselves as too small to deal with השקפה issues. Most had never even read the major works of Jewish Philosophy like Ibn Gavirol,  the Guide, or Joseph Albo. [For this reason I made it a point to get some background in the Guide and Saadia Gaon and basic world view ideas of the Geonim and Rishonim. Their world view is very unlike  you could imagine.]

And there is no indication that anyone after the Ari was anything within light years of the Ari. They have emotional appeal, but nothing as far as objective reality goes.

In any case we have a whole set of problems that have not been addressed very well. The nature of delusions, the nature of pseudo tzadikim, and the urge to worship pseudo tzadikim. The best I could do to get anywhere in this was to study different groups like Hindu cults and hope that that would give me some insight. As far as I got, still did not seem to matter much. No one in any case was really willing to listen. In any case, because these issues are not resolved,the best thing is to get the basic set of medieval thought, the Guide of the Rambam, Saadia Gaon's Emunot VeDeot, Abravenal, Joseph Albo, Crescas and get a decent idea of what Torah teaches in terms of world view. [It is not worship of tzadikim for one thing. But there is  a lot more to it.]

See Steven Hassan Escape from Cults

Bait and Switch is what he identifies as the major cult characteristic and this in fact seems to be the case. The hiding of the actual beliefs. First draw people in by seeming Kosher and then switching.
As for my own study of cults I found Steven Hassan helpful.

There is a simple test for cults. There is an objective change in character that can be seen.
When one joins a group like a Lithuanian yeshiva the change in "Midot" [Character traits for the better is obvious to all.] When on the other hand one joins a cult the change in character is also obvious. Who can's see how people's traits change for the better when they join a place like the Mir or Chaim Berlin? And places and groups that worship some tzadik. The deterioration in character is clear even to people in the group and takes effect almost immediately. And this has nothing to do with what you think of the tzadik. For all you know the tzadik might very well be a true tzadik. Still the effect on people's character is unmistakable. It is not necessarily that they become bad people. But their character changes towards something ugly. Some undefinable ugliness takes over their personality. Or in other groups some strange kind of cruelty and sadism  enters into their souls.

I should mention I did a lot of reading on this but I do not feel comfortable in going into detail about the Sitra Achra [the Dark Side]. I would hope that my warnings here should be enough.

In any case this is no more  a matter of discussion. Once the idolatry became clear it should have been time for action--decisive action.