NEVER feel guilty about buying yarn! It makes you feel good to feel it, imagine what you'll make with it.... as you said, the only place it hurts you is your wallet. Does your DH hide his hobby from you? Probably not. My DH's hobby is raising racing pigeons. I don't complain to him about the costs involved and he never says an unkind word about my yarn. Our only concern is that I'm running out of room for all my totes full of yarn (laugh). I never hide my yarn from him.
Shelagh