Downgrade Without Being Hit With a Penalty

Got a card just for the points reward, and have had it for nearly a year. Should I downgrade/cancel it before the second annual fee is billed or if I downgrade/cancel it after the bill hits, will it be refunded? Heard that Chase sometimes voids your points or hits you with some sort of penalty if you do it too early. I also already have a freedom unlimited card