Short version: Because when the average person hears "furry" they either think of the sexual aspect of the fandom, or those who the media has labeled freaks because they actually feel they're more animal than human. That's why admitting your a furry feels akin to admitting you're gay.
Currently, the only person in my non internet life who knows I'm a furry is my gf, and she's wonderfully cool with it. I suspect she's at least slightly furry herself, though I highly doubt she'd call herself one. Besides her, there's only one other person who knows me by name that knows I'm a furry. I know my family, at best, wouldn't understand it, and I know one uncle who wouldn't accept it. He's called them freaks once before in my presence and I had to remind myself that he didn't know he was also talking about me. First time I've ever had to truly force myself to laugh along with someone.