My Virgo won't let go of his ex...

I met my Virgo about 2 years ago through a mutual friend. He was in a relationship at the time and I had just broken up with my bf of 2 years and had been through some very heavy stuff and wasn't looking for love at the time. He found me on myspace and pursued hanging out with me "as friends." from the time we met until we hung out the first time he broke it off with the girl he had been dating for a year. Well anyways one day I just called him up and said how about that game of rummy? I went to his apartment played rummy with him and he then took me to dinner later that night and when I brought him back home I didn't want to leave him! I think I had fallen in love with him that day!!! He invited me in and we spent the night having the most amazing mind blowing sex I ever had. We continued this friends with benefits thing because i wasn't ready to jump into another serious relationship and I told him he could have sex with whoever he wanted. I am a capricorn woman so maybe I was testing him because I later found out his ex and he were still sleeping together and it really upset me. I told him it was her or me and that the whole open relationship thing was not something I could handle and that I wasn't sleeping with anyone else. Well 2 years later here we are we have been through a lot and I know we both love each other a great deal, but there have been a few occasions where we were fighting or broke up for a short time and he slept with this same ex again. Trust me when I say I have been no angel but I just don't know what to do if I should just let him go or what? Every time I try and and go and move on it never works and we end up back together because we can't just be friends and we can't seem to stay away from each other. He is literally the best sexual partner I have ever had and I know that if we could both just grow up and stop betraying each other's trust we could really be happy. I did something recently when I found out about him sleeping with her (AGAIN) that he just can't seem to let go. It has been 4 months and I still hear about it everyday! Is he ever going to let it go Is it too late for us? Have we both hurt each other too much? SHould I just let go? I just love him so much but sometimes I think he just doesn't want to see me with someone else and that is why he never lets me go completely? I wish I had all the answers....