Sometimes people get involved in books that are pseudo Torah and this slowly draws them away from Torah. And then they get to be experts in pseudo Torah and then get called תלמידי חכמים Torah scholars, though they can't learn legitimate Torah.

The Gra I think was advocating a kind of system in which people would learn Torah all day. And in his commentary on Pirkei Avot he says that learning Torah is a mitzvah in itself even if one does not keep anything at all--zilch of what it says. This is on the Mishna  [ch. 5] that one who goes to the Beit Midrash to learn but does not keep Torah gets the reward for learning.

In the Lithuanian yeshivas today that are built on the model of the Gra you can see this attitude played out in life.

The Gra did not write Musar, rather he wrote commentaries on every part of Torah. So you can't really find direct statements of what he holds on lots of issues. But you can see something of what he must have been thinking in the writings of his disciple Chaim of Voloshin. And in his writings you find this statement דבר מנוסה כשישכים אדם בבוקר ויקבל על עצמו עול תורה באמת היינו שיגמור בליבו שלא ישמע לשום אדם, ולא יבטלנו שום טירדה, אזי יצליח ביום ההוא בתורה. וכפי גודל ההסכמה ותוקף הקבלה כן יסירו הטרדות
והביטולים ממנו

The literal translation of this is this: It is a tested fact that when one gets up in the morning and accepts on himself the yoke of Torah in truth, that is, he decides in his heart that he will not listen to any person and he will not allow anything to distract him, then he will succeed in Torah that day. And according to how strong his conviction is and the power of his commitment, to that same degree there will be removed from him the distractions and everything that is wasting his time.

Personally I have found an enormous amount of obstacles in reaching this goal. I still spend practically all my day not learning Torah. And I don't have much of an excuse either.
There is very minor part of my day that I try to do things that might be considered as reasonable excuses for not learning Torah. But if you add them all up they would amount to minuscule amounts of time. Most of all the obstacles are mental and physical.

There are in my mind lots of reasonable things that would require me or others to stop  learning Torah for  a few minutes in order to attend to these matters, and then return to  learning Torah. But what seems to happen is the entire idea of  ביטול תורה (wasting time from Torah) gets thrown to the winds.

While I agree that one should learn an honest profession, get a real job, and not depend on charity. But what happens is that Torah is thrown out completely. Guys are  capable only of concentrating on one thing at a time. If it is Torah, then it's Torah. If you try to combine that with something else, the "something else" becomes primary, and the Torah get shoved to a very secondary place. [I think the Gra was defining Torah in  a limited sense. We find the Rambam also saying this remarkable statement, "Just as anything that one adds to the Written law is not Torah so anything that one adds to the Oral Law is not Torah." Today it is common to teach things that are not a part of the written Law or the Talmud and to say that one is teaching Torah.

Part of the problem is mental. I have  a hard time along with many other people in seeing the point. It takes a large degree of faith to believe  that one is accomplishing anything at all by learning Talmud which at first glance is about as interesting as the New York Phone Book.
It is tempting to learn other things and call them "Torah" --even if for no other reason that they are more fun.

But there are many other reason to not learn Torah. It does happen that just when one starts to learn that the obstacles gain in strength. And people come up to him and say, " Let's go and do some mitzvah." You can't see the effect of learning Torah until a lot of time has passed.

It is funny that there are so many distractions that seem  perfectly legitimate. Sometimes people get involved in books that are pseudo Torah,  and this slowly draws them away from Torah.  I see this all the time. Then they get to be experts in pseudo Torah, and then get called תלמידי חכמים Torah scholars, though they can't learn legitimate Torah.

So what I suggest is this: learn Torah. I can agree with an hour a day of learning a profession in order not to have to use the Torah for money as is so common. And a certain amount of Physics and Metaphysics is also important to the Rambam. But that is not an all encompassing excuse to spend the whole day in extracurricular activities. And drop all pseudo Torah. You know exactly what I mean. I don't have to spell it out.

The justification for this is this learning Torah produces a type of consciousness. And this consciousness is the source of one's deeds.
In any case I got involved in extra curricular activities a long time ago and so it makes plenty of sense to me that I have found so many problems in my life. I think it all flows from ביטול תורה--wasting time from learning Torah.