Should I just walk away from this tired some relationship?

A few weeks ago, this guy i used to be off an on with for about 2 years ago, involved with recently got back in touch with me. We stopped speaking a about seven months ago, so he called me up, and He told me he missed the sex, and he misses being with me since I'm so much fun...so he gave me his new number and he tells me straight up, he doesnt want a girlfriend, and not looking for one, and that he probably will not have a girlfriend until his son goes to college, which is about five years from now, his son is 12 now.So a few days later, we set up a date to get together this Saturday(yesterday).So I text him, asking him to go to this new restaurant, and I would treat.He tells me hes with his son, and there at the movies, and he will be out late, around 12-1 in the morning.So he calls me at around 10 and hes like lets get together and have sex, and he asks me to drive to his house, pay for the hotel room, and tells me he will not spend the night because I am married, and he isnt even comfortable being with me since I am married.I explained to him, that yes, I am legally married, but it was to help my friend with an immigration situation, which I am not proud of, but it is that, he goes on to say, its immorral, and he doesnt like the fact that I am married, and that he shouldnt even be with me period, so his phone disconnects, and doesnt call me back, I tried texting him, and hes just ignoring me..i'm confused...