Uninvited guests

I really want an impartial answer to this, and I am going to be as objective as possible.
My wife and I have been married for six years. She has grown children from a previous marriage.
Her Son has a habit of stopping by the house uninvited. The last three times, I asked him to please call before he comes over, just as a matter of common curtesy. He always smiles and says "Sure, no problem", but turns around and does it again. The next time I get a little grumpy about it and beg him to just call first. Again, he say's OK.

Today I am laying on my couch in my underwear watching tv and glance out the window to see him, his wife and his baby walking up our driveway. I throw on my pants and open the door. "What the hell man?" Oh, the battery is dead on my cell phone..." So I get full blown pissed off and tell him that he NEEDS to call and that it is a matter of common sense. He counters with "But I grew up in this house and I am here to see my mom!" To which his mom (my wife) chimes in "He can come here any time he wants and he does not ever have to call and he is my son and this is half my house, so you will need to aplogize to my son!" To which I say; "There is no way I am going to aplogize. I have asked him at least three times to call." The son gets very upset and says "Fine, then we'll never come back to visit" and storms out, leaving me feeling like a jackass. Now my wife is angry with me, my step son is angry with me, and suddenly I wonder if I am right or wrong. Should relatives be above the law, so to speak? Children and parents can stop by their respective relatives house without a call or invitation? Is that the way it works? I was brought up that you always call before you stop by someones house, but maybe that was just a weird family rule (?) What gives?