Immigration panic comes in waves. Data shows who worries most, and when