Kate, imo, your situation applies to the "allergy" category, although I realize what you are talking about doesn't constitute an allergy in the classic sense. If eating certain foods causes illness, then being "picky" is totally understandable.
But I've been to people's houses where some of the guests (usually relatives) were turning up their noses wanting to know what the "green stuff" was in the casserole. Okay, so if a person doesn't like "green stuff", then don't eat it, but there is no reason to act offended because it made an appearance on the buffet table.
Best regards, Michelle in Nevada