Not all secular learning is secular. Even though Rav Nahman and many others  say to stay away from secular studies, still not all that is listed as secular is in fact so. On one hand, some are clearly man-made and not a part of Torah, however there are wisdoms that include the wisdom of God. So you find Rishonim (mediaeval authorities) that include Physics in the category of the things one is supposed to learn every day. And among those Rishonim are Ibn Pakuda. the author of the Duties of the Hearts.-

So while in the Mir in NY, one was not allowed to attend Brooklyn Collage  while at the Mir, still when I asked the rosh yeshiva [Rav Shmuel Berenbaum] about this policy, he said one can learn Physics for the sake of making a living. But as deep a thinker he was, I think that this distinction  he did not make.