How complex an AI Fortune Cookie wrapper could possibly be? Solving LLM determinism with "Salt"

You could think, AI fortune cookies? Pfft, that's the simplest API wrapper one could possibly imagine. Just instruct some GPT model to generate "creative" results, wrap it in a nice UI, and ship. That's so criminally simple it's just lazy to develop something like that, so boo to you, FriskyFennecFox! Right? Nope.

The critical flaw, >!aside from my extreme perfectionism,!< as it often goes with this modern day gold rush, is in the "AI" part.

On paper, it might seem that by hooking your API to an LLM, you automatically get an infinite amount of fortune cookie messages and become a new worldwide fortune cookie magnate, never having your users receive repetitive messages compared to a traditional pre-defined list. After all, you told your model "creative", so it got to be! The second you deploy it, it's just a matter of fact that your business email will be instantly filled with B2B requests to grab a piece from your endless bucket of data!

But the instruction-tuned LLMs are, essentially, deterministic. In their "assistant alignment", they always have constraints set by their training data, which encourages one set of tokens and discourages the other. One could pick a non-overfit model and write a complex prompt & play with the samplers to force it to write "creatively" and generate how much, 5, 10, 100 creative messages? Let's settle at 1000.

But what happens once you sample that batch of 1000 AI-generated messages to review the quality and objectively check how "creative" your model of choice actually is? You'll inevitably notice that some tokens in those fortune cookies repeat way too often:

• "Bright", "Joy", "Success"
• Random objects ("River", "Mountain")
• The semicolon and the infamous hyphen symbol

What's even more crucial than single tokens is that you will catch your model generating not only the repetitive words but also follow the same "lazy formula":

• "A joyful path awaits... ...journey of learning and discovery"
• "A gentle spark... ...the path of joy and harmony"
• "A journey... ...new light and joy"

Notice not only the repetitive words but the way it ends the fortunes with "something1 and something2", that's the laziness that trips a lot of creative writing use-cases even when a model writes fire at the first glance.

At one point, the user's mind will notice the repetition, the "magic" will be lost, and they won't bother opening the app ever again because they already know what kind of fortune cookie message they're going to receive.

There are a couple of solutions to this problem. You could:

• Build a dataset of fortune cookies and finetune a model from scratch to serve from your own hardware. But, after a couple of experiments, I discovered that the dataset needs to be truly diverse. If you try using synthetic data, you'll simply introduce the same bias you were trying to escape to your custom model.
• Use a form of RAG to pull the data from the logs of the previous generations and increase the repetition penalty sampler to enforce a model to create a different output. This is a conceptual idea, but it requires a more advanced backend shenanigans, increased network activity, adds a point of failure, and, more importantly, substantially raises the pricing per call by increasing the input length of each call.
• Go wild and deploy an untuned model, praying your regex would work reliably at scale and your moderation endpoint or a "rewriter" model is 100% robust in its ability to filter out sketchy or questionable messages before they reach the end user, accounting for the added latency with the "bounced" generations.

But there's a simpler solution, I ended up using a technique of an "entropy salt".

Essentially, it introduces randomness by guiding an instruction-tuned model to subtly account for a certain string without sabotaging the main instruction.

Since LLMs are about ^(numeric representations of) human language, the "salt" in this case can't be a random string of letters or numbers you'd expect from a salt in, say, cryptography. An LLM processes English, so the "salt" must also be in a human language.

I decided to use quotes from novels as they felt the most fitting to be used in conjunction with fortune cookies. My initial choice fell on random book quotes, but, after messing with Project Gutenberg books, NLTK, and Markov chain, I settled on extracting such data from a base LLM instead.

Here's what I did:

1. Prepared a local API with the raw weights of a base model that had no SFT post-training or excessive multi-stage pretraining that might've encouraged narrow biases;
2. Prepared the "prompt", essentially a string appearing as a JSON dataset entry for a list of fictional quotes, ending with " for the LLM to fill the rest of the quote;
3. Prepared a Python script to make 10.000 API calls to create a raw dataset of such "nonexistent quotes" before filtering out those which are either too short or too long;
4. Prepared a Python script to perform a second run over the raw dataset, using an assistant model to fix logical inconsistencies and formatting errors, as well as and rewrite & filter out the unsafe quotes*.

* You never know in what kind of an emotional state a user who comes to a service with close ties to the wellness and spirituality niche might be in, and you absolutely do not want to harm such a user with a "bad pull" if a model would accidentally misunderstand a dark quote as an instruction to generate an "unfortunate cookie".

The resulting dataset ended up having 7000+ lines of such "quotes".

Now, in the production, these lines are randomly selected with each API call before getting injected into the instruction. And yes, while a certain bias toward some words is still present, it has dramatically increased diversity of the generated messages without sabotaging stylistic consistency.

It targets the wellness crowd, but I will gladly welcome a comment from the tech-savvy enthusiasts as well:

- https://cookieai.co

If you want to experiment with this prompt injection technique or just want a dataset of random quotes, you can grab it on HuggingFace:

- https://huggingface.co/datasets/FriskyFennec/synthetic-quotes-v1