It is usually understood that when there is an argument among Rishonim [mediaeval authorities] it does not make sense to say one or the other was right. You might do like the Beit Yoseph that you go by three only. That is the Rif, Rambam and Rosh. When two of these three agree to anything, that is the law. Still that does not make the other wrong. And when you can not find a consensus among these three, then you go by the majority of Rishonim.

So in the case where many of the Rishonim that hold Physics is a part of learning Torah, how would you decide that? Some hold yes, and some hold not. Since the Rambam is clear, and the Rosh and Rif do not openly discuss this, it seems clear the law is like the Rambam--especially after many rishonim go with the Rambam in this point.
[You might add the fact that both the Rif and Rosh say that "outside books" in Sanhedrin are not what people often think are "outside books". They are not books of Math and Physics. "Outside" means giving explanations not from the sages of the Mishna or Gemara or Midrash.]


There is an aspect here of experience also. It does not to me seem that learning Torah alone with these two added things Physics and Metaphysics [as the Rambam phrases it in the Guide] really leads to human perfection.