Conflict of interest?

I'll try my best to explain my dilemma while trying to stay anonymous.

I currently hold a manager position in a Fortune 100 company. I'm not seeing opportunities to grow in my department and have already decided to look for opportunities in another department.

Today a recruiter reached out to me about a very interesting position. It is a Director role in a smaller company, but still fortunate 500.

My dilemma is that this recruiter works for the same agency we use to recruit candidates. I personally have hired multiple people thru them.

My concern is if I apply to this position, it can leak to my manager. They're a smaller agency, and we are one of their major accounts, and their loyalty will be to the people signing the checks.

What are your thoughts? Is it worth the risk?