The main engine of yeshivas in NY is the idea ביטול תורה כנגד כולם. Lack of learning Torah is equal to all the other sins put together.
It is not the idea that learning Torah is equal to all the mitzvot put together. If learning Torah was just a nice mitzvah there would be little reason for yeshiva.
But this idea that lack of learning Torah is equal to all the other sins together means that Torah is an obligation on every person.
And the idea that lack of learning torah when one is able to learn is a sin has a good source in the gemara in Sanhedrin כי דבר השם בזה הכרת תכרת הנפש ההיא מקרב עמה זה מי שאפשר לו ללמוד ואינו לומד.
This would be the reason why I myself went against my parents wishes and went to yeshiva instead of to university. I felt learning Torah was that important. Still in hindsight I  see that my parents were right and if I could go back, I would have learned half a day in the yeshiva, and spent the other half in Brooklyn Collage.

I know there are different opinions about this issue. Some people think that one should learn Torah all the time and that is that. That is in fact the general approach of Lithuanian yeshivas in Israel. In fact, in Israel if one works and learns he is considered a second class citizen in the Charedi world. Forget about decent shidduchim for his children. People won't touch him with a ten foot pole.
And based on the statement in the Talmud about the importance of learning all the time it is hard to argue with the Israeli approach.
I don't have a clear resolution to this matter, but I think that a possible solution goes like this: If you are learning Torah and you don't let go for any reason, then there will be help from heaven that you can continue to learn. But if you let go, even a little bit, then you will not be able to get back to it. And if you try to get back to it after you gave it up --it will blow up in your face. It won't be real Torah you will get back to, but some false pseudo Torah. [I can't explain this. It is just what I think I see happens.]
I can' answer this contradiction and I don't minimize its importance. But I can minimize the area of conflict.
I claim there is much less of a controversy here than people think. Litvaks traditionally had a side learning project. And we know the Rambam held that one must learn Physics and Metaphysics. I think that areas outside STEM subjects in fact should be shut down in universities. I can't see any good in any of the social or humanities departments  in most colleges.