Uncomfortable Charity

Why does the act of charity, in this case, the transaction initiated by a beggar or panhandler, feel so uncomfortable to me? Mental recriminations if I give, guilt if I don’t. Perhaps it is because I don’t know the protocol, the expectations, and so I’m worried about an inadvertent transgression. But it isn’t that hard to act, to find a way to overcome my anxiety and hesitation to do something small.