LLMs and Playtesting

I'm fully expecting to get reflexively down voted due to the anti AI sentiment in this community. Full disclosure, I share that sentiment for the most part. I however use LLMs quite a lot for work and as such have expensive plans and ample reasons to get comfortable with them and their capabilities.

​

The question I'm coming here with is, "Do LLMs have a role in Playtesting?" Before you say "No" please note that I of course believe that; obviously human playtesters are better.

​

There are just some unfortunate limitations with human beings, in that most of them have better things to do with their time than playtest my jank ass system before it's reached a certain level of polish.

​

Spreadsheets and probabilities cannot really calculate fun, and one person's perspective can't really catch everything. Let alone most things.

​

I've experimented with LLMs in the design process as a second set of eyes and concluded that they're next to worthless for anything other than formatting my drafts to markdown formats and catching inconsistencies between terms. I have a habit of defining things more than once, so it's helpful for that. But in terms of real game design sense, it's worthless. So that's out the window.

​

What I'm referring to is using agents as players.

​

I set up an experiment and ran 3 agents in a chat log and acted as the DM for a few rounds. I provided them a quick reference rules sheet and an MCP tool to roll dice so they couldn't fudge.

​

The results were middling. The bots misunderstood some mechanics but made decisions comparable to how a normal player might act under the right circumstances. They had a propensity to really smooth over things not working just to play along. Overall, not terribly helpful. Spent about 2 hours on the whole experiment and came here to ask

​

has anyone managed to get something like this to work?

​

If so, what methods did you use? What were your results? I know people don't like AI, that's fine. There's lots of good reasons not to, and I'm not here to argue about it. Way I see it, is it's a tool that can maybe help me get my system more polished the next time I can wrangle together 2-3 players for my next playtest.

​

It's frustrating when you get people to set aside time to do you a huge favor by indulging your creation and then it fails in some major fashion that makes the experience a chore. I respect peoples time for one, and for two I just find it quite hard to actually wrangle a group of playtesters. In the spaces where RPG design is discussed, everyone has their own system they want to playtest, and outside of that you're bothering your friends and associates.

​

So, I open the floor up to you all. On all these subjects. I am prepared for down votes as well.