Pedestrians crossing on green for cars but not for them

Is it just me or do people not understand the purpose of the crossing button and the red hand signal? In so many instances cars get green light, hand signal is red and people start crossing the street. If I am third, fourth car waiting to make the turn, most of the time I won't be able to because time is up. Pedestrians effectively are crossing on red, breaking the law, putting themselves in danger and creating more traffic jams.