I agree with everything you said (How often do you hear that?) I'd like to add, however, that sex isn't always just sex.
For some people (me) monogamy is a way of showing a loyalty to the feelings of your partner and vice-versa. Neither or us have any moral judgment about it. Rather, it's the thought of how emotionally complicated it would be if we let a 3rd party into what's been our own private experience for over 15 years.
We watched Big Love together and at the end of the first episode, my partner Patrick blurted out "Okay, I will never want more than one wife." I had to laugh because as much as more sex with other people might be attractive, it usually comes with baggage. It's the baggage we don't want.
Granted, sex with the same partner changes over time. And the rush you get from being with someone who is totally hot for you because it's all new is fabulous. Who doesn't want that?
The thing is, that is never all you get.
Remember the movie Fatal Attraction? Enough said.