Although I had sex with him once and I didn't know him from a long time, I have feelings for him?

If you can't handle the fact that all he wants is a physical relationship, you did the right thing. And yes, it's pretty common to fall for your first. I've seen it happen a number of times.