The obvious thing is that in the Torah world, we do have books and a lot of them are valuable.
I would think that there is a difference between the oral law and the written law. But that is not the answer he gives. He says the difference is between books that deal with understanding how to keep the commandments of God as explained in the Gemara according to the foundational principles of the Talmud which is OK and books on hashkafa [the world view and philosophy of Torah--e.g. understanding the reasons for the mitzvot.
And he has a point. I can see important value for books on orientation like the Guide for the Perplexed by the Rambam and the Horev from Shimshon Raphael Hirsch, and the books of Rav Avraham Kook. But mainly books on Hashkafa seem to me to be talking nonsense.
What is wrong with books of Torah hashkafa [world view]? One problem is the question who is qualified to write about what the Torah holds? Who is qualified to even have an opinion about such a thing? Only someone who has read through the material and knows it well. Someone like for instance Moshe Ben Maimon (Maimonides). If anyone is qualified to have an opinion about what the Torah hold surely he must be at the top of the list. And to our great happiness he actually wrote a book explaining what the world view of the Torah is. You would think that people would be overjoyed. But no. People don't want to hear what he has to say because they have their own opinions about what they think the Torah ought to be saying. And they write they own books of nonsense and tell you not to learn the Rambam's book because it might confuse you and take you away from their own views which are contrary to those of the Rambam.