Sex outside of marriage is not ניאוף-adultery. It comes under the category of פילגש concubinage.
There were a good number of people in the Old Testament that had concubines. One well known such person was Calev ben Yefuna (the friend of Joshua, the disciple of Moses). [כלב בן יפונה ] He was not considered a sinner because he had a few wives and few concubines. In the Five Books of Moses he is praised with a kind of praise that is not applied to anyone else in the Torah, וימלא אחרי השם "He was filled with God." People might not think this is good but they cant call it "sin" in Biblical terms. And when pastors do so they are just displaying their lack of education.
I forget where the list of Calev's wives and girl friends was. I think it should be in Chronicles one where it lists the people from the tribe of Yehuda that came into Israel. One of them was Calev because Moshe told him and his buddy Joshua that they would merit to enter the land of Israel.
There are lots of sins in the Bible but typically they have nothing to do with what pastors call sin. And what pastors call commandments are often highly sinful.
If you want to know what the Bible calls a sin just open a Five Books Of Moses and take a look at when it says, "God spoke to Moses saying speak unto the children of Israel say unto them thou shalt not do such and such a thing. and if you do so this and this is the punishment." There are lots of examples. You might say that if God says not to do something that has to count as a sin. There are of course no pastors in the world which do this simple process. It is really not complicated. But instead they have to fish around for other things to call sins.
On the other hand I can see why people just don't jump to the Five Books of Moses. They are afraid of groups that claim to be following these laws. And that is justifiable. people that claim to be teaching Torah are mainly "Torah scholars that are demons תלמידי חכמים שדיים יהודאים . So that leaves anyone that wants to keep the Law of Moses in a bind. There is no where to go learn how in a practical sense to keep the Torah. If the people that claim to be teaching it are demons, that limits the options.
In real life, we encounter people that are supposedly teaching Torah that are demons. I think I have probably met quite a few of them.
In any case, what you could do is to simply learn the Five Books of Moses on your own, and then just go through the Mishna and Talmud on your own to discover how to keep the law in a practical sense. That is what I do. If there was an authentic Lithuanian yeshiva near me I would go there but that is not an option for me now.