The difficulty in learning Torah is often the religious themselves. In particular I noticed that people that make money off of Torah are are a problem.
Rav Nahman of Breslov brings up this problem in a few places in his LeM. But one place in particular that I recall says that when one wants to come into the gates of holiness, from heaven there is placed someone that seems like one who fears God to stop him.`
At any rate, this problem is wide spread and pervasive./It is almost a guarantee that anyone who makes their living by using Torah (to make their money) is from the realm of Evil.
I am, however, not saying that this is any kind of reason not to learn Torah. Rather my point is that one ought to learn Torah the right way,-- or not do it at all. Either (1) for its own sake, and not use it for making money as is done in kollel; (2) or not to do it at all. It is like any task. Do it right, or do not do it at all. But what ever you do, do not do it wrong.