Sleeping with friend's Cancer brother and so confused about secrets.

Hi I am a 23 yr old Gemini female and I'm sleeping with a 21 yr old cancer male. I'm in a very difficult situation because I am best friends with his older sister. I have known the family all my life and am extremely close to them all, I'm the 2nd daughter to his parents and my children are grand children to his parents. No one knows we are sleeping together. We both know it will cause huge family fights if they found out. Our encounters started just over a year ago after he made the first move when he was giving me a back rub. We are not in a relationship but have casual sex. We have to be careful not to draw any attention to ourselves so no one finds out. When we first started I spent a lot of time at his place (both siblings still live with their parents) his sister and I got into a fight because I would spend time with her brother while I was there as well. It was at that point that we decided to not see each other as often. We then started doing movie nights as a group with a few other friends as an excuse to spend time together. Some times at their's but more often at my place. We all get along well. In the past month he has been asking his sister to come over with her to mine to help with some stuff ( because I don't have a car at the moment) the brother and sister have stayed over 2 times in the past month but he has made excuses to come over at least 5 or 6 times "for the drive". We never touched each other unless we were planning to sleep with one another that night but lately when he has been coming over more he has been touching me on the bum or leg and giving me quick stares when no one notices so I'm a little confused to what he wants and I don't want to ask and look like a fool. My best friend says he likes a mutual friend of ours and I feel a bit jealous and threatened even though I know he won't touch her because she is still a minor. I know my feelings have developed more but I can't tell with his actions if his feelings have to. We did talk about it about about 6 months ago and he said he isn't interested in a relationship just sex. He hasn't dated in 3 yrs and I'm the only one he has slept with since is last psycho girlfriend. The sex is fantastic and he is very affectionate when were alone, we flirt and play fight and tease each other too. I have fallen fall him and he knew that 6 months ago and we did stop for a little while but then we gave in. Have you got any advice on this situation? How can I tell through his actions and star sign if there is more for him?

Ps the reason we never sneak out is because he lost his license for 2 yrs a few months before we started having sex.