tell the truth? PLEASE HELP

I have been with my boyfriend for about four years. Everything is wonderful and we often talk about getting married and children. The only thing is, I lied to him about something really silly when we first met. I told him that I got a ridiculously high score on SAT's (which I did not take) and applied to Yale and was accepted. I guess I told him that because I figured we would never really pursue a serious relationship and he would never find out. Well, it is four years later and he still brings up the story of how I got into Yale, as he is extremely proud to be with someone who could've, but chose not to, go to Yale. It is the only lie I have told him and it is a constant worry for me... that he will bring it up to my family and they will have to unknowingly out me and my outrageous lie. What do I do? Do I just come clean? How would I do it?