Happy Friday! We've been working on a system to evaluate the quality and performance of MCP servers. Having agentic MCP server evals ensures that LLMs can understand how to use the server's tools from and end user's perspective. The same system is also used to penetration test your MCP server to ensure that your server is secure, that it follows access controls / OAuth scopes.

**Penetration testing**

We're thinking about how this system can make MCP servers more secure. MCP is going towards the direction of stateless remote servers. Remote servers need to properly handle authentication the large traffic volume coming in. The server must not expose the data of others, and OAuth scopes must be respected.

We imagine a testing system that can catch vulnerabilities like:

* Broken authorization and authentication - making sure that auth and permissions work. Users actions are permission restricted.
* Injection attack - ensure that parameters passed into tools don’t expose an injection attack.
* Rate limiting - ensure that rate limits are followed appropriately.
* Data exposure - making sure that tools don’t expose data beyond what is expected

**Evals**

As mentioned, evals ensures that your users workflows work when using your server. You can also run evals in a CICD to catch any regressions made.

Goals with evals:

* Provide a trace so you can observe how LLM's reason with using your server.
* Track metrics such as token use to ensure the server doesn't take up too much context window.
* Simulate different end user environments like Claude Desktop, Cursor, and coding agents like Codex.

**Putting it together**

At a high level the system:

1. Create an agent. Have the agent connect to your MCP server and use its tools
2. Let the agent run prompts you defined in your test cases.
3. Ensures that the right tools are being called and the end behavior
4. Run test cases many iterations to normalize test results (agentic tests are non-deterministic).

When creating test cases, you should create prompts that mirror real workflows your customers are using. For example, if you're evaluating PayPal's MCP server, a test case can be "Can you check my account balance?".

If you find this interesting, let's stay in touch! Consider checking out what we're building:

[https://www.mcpjam.com/](https://www.mcpjam.com/)

MCP evals and pen testing - my thoughts on a good approach