Transform selection box without moving opposite border

This is an infuriating behaviour that costs me a lot of time when, for example, cropping a video. I place one border of the selection box where I want to crop my video on that side. Then I want to move in the other border of the selection box to where I want to crop the video on that side. But moving one side also move the opposite side, so now the side that I placed first isn't where I want it to be. Is there a way to just move one side without the other one moving as well? Alt, Ctrl or Shift don't work.