This comes from an insight of Rav Nahman about learning in conversation 76 on the conversations of Rav Nahman. But I combine it with the few Rishonim that agree with the importance of math and physics. And I say that the idea that only talented people can learn math and physics is wrong. I say with the way of learning that Rav Nahman explained in the conversations 76 even a block of wood can know math and physics.
Rav Nahman said --"to say the words in order and automatically one will understand, And even if he does not understand immediately eventually he will understand.''
But he did not apply this to Physics but rather to learning Torah as is brought in the Talmud Tractate Shabat page 63. Still I learned late to discover that some rishonim hold of the importance of Physics and Metaphysics as you can see in the Mishna Torah chapter III:" The subjects discussed in the first four chapters of the Mishna Torah are in the category of learning Gemara."
But are these the only things to learn? Well, to the Gra the seven wisdoms are needed for understanding Torah {Trivium and Quadrium) but besides that? I would say besides these everything else is Bitul Torah
[I did read in high school and later in Shar Yashuv in NY but i can not tell how much of that did me much good. Maybe it did or maybe not--. The books were what was required in an average American public high school. But most of my time was spent on the violin and piano. I did not get into Gemara until my first year after high school at Shar Yashuv [3.5 years there and then another 3.5 years at the Mir in NY. But there I did mainly Gemara. ]