Is fishing hard or do I suck?

Haven't been able to actually catch anything with a giant shadow, they always manage to escape. Also everything I catch seems to be considered "tiny" adn Diamant says something like "fishing isn't all about results"

Am I doing something wrong? I always try to move the L stick in the opposite direction to the fish, but it doesn't seem to help