In this he was depending on statements in the Talmud (Yerushalmi Peah chapter 2). Also there is considerable support for this idea in the Zohar. The disciple of the Gra goes into this in detail in his book the Nefesh Hachaim.
. The fact is the Gra has support from the Talmud. There is a Halacha that when there is a mitzvah that needs doing and it can't be done by someone else one should interrupt his learning of Torah to attend to that Mitzvah. And praying to God to be saved from sin and to be drawn towards His service is a mitzvah that can't be done by anyone else.
The Gra (Eliyahu from Vilnius) is someone that is not charismatic. You can't get excited about the Gra. But it is possible to get excited about the Torah.
I sadly dropped my extent of involvement with the Torah. Sure I would keep on learning to some degree, but mainly my time was taken up by kivrei tzadikim [graves of saints] and saying lots of tehilim [psalms] and the like. Anything but Torah. And I discovered and odd fact. That when one drops Torah he can't just pick her up again when he pleases.