I believe most self-esteem issues come from one of two places: people who say stupid things without evening realizing how devastating they can be (like your BF's comment) or from comparing ourselves to others.

My sister always says "Comparisons are odious." and I agree.

They always leave us either feeling a false sense of superiority OR a false sense of insecurity--both are false because there will always be someone better/worse than you no matter what category you compete in.

So listen to yourself, be the best you can be and forget the rest. It's just noise.