What the others have said: I donate to charities, instead. I also just never have actual money on me these days, so there's rarely anything on me to give.
I do always feel awful when I walk past or say no to someone, but the stories you hear about a lot of this money going to sinister groups is enough to deter me.