A simple script to embed static sections of prompt into the model instead of holding them in context

https://github.com/Void-07D5/LLM-Embedded-Prompts

I hope this isn't too late for the contest, but it isn't as though I expect something so simple to win anything.

This script was originally part of a larger project which the contest here gave me the motivation to work on again, unfortunately it turned out that this larger project had some equally large design flaws that weren't easily fixable, but since I still wanted to have something, if only something small, to show for my efforts, I've taken this piece of it which was functional and am posting it on its own.

Essentially, the idea behind this is to fine-tuned static system prompts into the model itself, rather than constantly wasting a certain amount of context length on them. Task-specific models rather than prompted generalists seem like the way forward to me, but unfortunately the creation of such task-specific models is a lot more involved than just writing a system prompt. This is an attempt at fixing this, by making fine-tuning a model as simple as writing a system prompt.

The script generates a dataset which is meant to represent the behaviour difference resulting from a prompt, which can then be used to train the model for this behaviour even in the absence of the prompt.

Theoretically, this might be able to embed things like instructions for structured output or tool use information, but this would likely require a very large number of examples and I don't have the time or the compute to generate that many.

Exact usage is in the readme file. Please forgive any mistakes as this is essentially half an idea I ripped out of a different project, and also my first time posting code publicly to github.