Perhaps we lack a right to engage in activities that reasonably appear to show an intention to harm or impose unacceptable risks on others. For example, I may not run towards you brandishing a sword, even if I do not in fact intend to hurt you. The principle also explains why we punish people for merely attempting or conspiring to commit crimes.
Thus, suppose that people who read the Communist Manifesto are slightly more likely than the average person to attempt the violent overthrow of the government. (This might be because such people are more likely to already have designs for overthrowing the government, and/or because the reading of the book occasionally causes people to acquire such intentions.) I take it that this would not show there is no on the face of it right to read the Communist Manifesto—though perhaps the situation would be otherwise if the reading of the Manifesto had a very strong tendency to cause revolutionary efforts, or if the occurrence of this effect did not depend on further free choices on the part of the reader.
But let us take the Koran. There is a strong tendency on the part of people that read it to take violent actions against Jews and Christians. This should be taken as good cause to throw it out.
Similarly I think we can show similar effects from that certain books of the cult that the Gra signed the excommunication on. That is they may not directly cause violence, but cause insanity.
In other words reading any book we have to assume is OK unless some kind of meme inside of it seems to cause people to swallow it and become violent or insane. It is for this reason that pharmacies are not allowed to sell poison under a label "Pain killer. Guaranteed to relieve all your ills." [The advertising would be absolutely true. Still it would be against the law.]