Moving on when you know there are fish

The other day I was fly fishing with a friend. We had gotten up early and staked out on a nice hole. After fishing for a while we started noticing an area of the river where we could see some nice sized trout hanging out and eating small bugs. We fished that area for several hours and watched the trout look at and reject most of the fly’s in our boxes.

All that to say, at what point in a similar situation would you have moved on to another hole? I had a great time and did end up catching one after dozens of casts, but would a more experienced angler have moved on after a bit? would you have done what we did and kept trying different flies until something worked?