You do not really know what is going on in the souls of other men. What do people gain by learning Torah in the kollel system? Sex--(i.e. a shiduch), money as their salary for learning Torah, power over others when what they want they claim is the authority of the Torah. Who is to say that they do not want these these things? Maybe they do. And if they do then that is Torah shelo lishma, Torah not for its own sake. And on Torah  not for  own sake we learn יערוף כמטר לקחי לשון הריגה "My teaching flows like rain" Flows is the same verb root as "kills". I.e., my teaching kills those who are not worthy. That means not only those that learn Torah not for its own sake but also those that listen to them.