I've never had a problem with it. I've been "fortunate" enough to know some extremely terrible people. My general philosophy is, if you're actively making life miserable for at least one other person - not accidentally or obliviously, but you are well aware that your actions are causing harm to other people and you simply don't give a sh*t - then you're taking up space and need to be taken off the board.
Example: When my parents split up, each of them picked horrible, worthless people as new partners. My mother chose a scumbag heroin addict who she'd been f*cking on the sly for a while; she wasn't big on the drug before that -she was typically a cocaine, alcohol and pill enthusiast - but once she was with him she became a full-blow addict and died within a year. In between keeping her doped up, he'd often get violent and abusive. The day she died, evidence suggests that he gave her the drugs, saw her overdose, panicked, went to work, came home early after about two or three hours, "found" her, and THEN called 911, by which point it was far too late.
It took everything I had not to end him after that, but I did tell him to his face, openly and repeatedly, "I can't wait for you to die." Which he did, eventually. I didn't get the full details, but at some point he dropped dead and I was glad.
My father made a similar mistake, and started tomcatting around with a rotating circle of crackheads and junkies. One of them physically attacked both me and my wife on numerous occasions, to which the police refused to intervene on the grounds of it happened in my father's apartment, and thus, "If he says it's okay then there's nothing we can do", which is bullsh*t but frankly, the cops were called to that place I don't know how many hundreds of times, I'm not surprised that at some point they just gave up. The final straw came one day when I was at work, and my wife was in our bedroom with the door locked, and Dad's girlfriend allegedly started banging on the door making threats. When my wife tried to go to the bathroom, the crazy bitch attacked her; they trashed the entire room, and at some point my wife managed to slash her across the face with a 13-inch ceremonial dagger, sending her to the hospital with stitches. I came home from work to discover all this; the cops did their job, but all the while were laughing about how "Somebody finally gave that crazy bitch what she deserved". We argued it down to a fine, but I told the bitch to her face, "If I was home and you pulled this sh*t, you probably wouldn't be standing here." She died a couple years later; again, I didn't get the full story, but I was glad.
There's been several others - so far, my track record is pretty good at this!
- but those two always jump out at me whenever anyone asks this kind of thing. I forget who coined the phrase, but the truth is, "If you think there's good in everyone, you haven't MET 'everyone'."
I don't believe human life has an intrinsic value. I believe whatever value you have is earned through your actions, deeds, and intentions, and you can "win" or "lose" points as you go along. Some people are absolutely worthless; I don't think it makes anyone a bad person for wanting those people out of the way. They're doing no good on this Earth and are actively making life worse for many others.