Help needed regarding Shadow Crystals not transferring

I've just reached Chapter 5 and am trying to get Chapter 4's Shadow Crystal to transfer but it's failing and I'm not sure what's going on.

I've been beating the main story on slot 1, but missed Chapter 4's secret boss which I beat on slot 2. However even if I swap around Chapter 4's slots so that Chapter 5's slot matches 4's Shadow Crystal slot on the main menu, I don't get the Crystal from the hole in Castle Town (even though I'm pretty sure that's meant to work).

Am I doing something wrong?