How does corpus size affect an LLM? Would one trained on just a book still be able to grasp the whole language?
I'm trying to understand how various factors affect LLMs. Specifically the size of the dataset they're trained on.
What would be the main difference between:
- A regular LLM (like ChatGPT) that's trained on the entire internet
- Same LLM but trained on a very small dataset, like just one book - harry potter
Would it still be as proficient at language, if not the knowledge?
Example: If I posed the question "How long did the COVID pandemic last?", would it still try to answer in perfect English but without the actual information, like "Ah, COVID, that pesky little poltergeist that's been plaguing the Muggle world for longer than a troll under the Whomping Willow!"
Or will it just be gibberish because one book is not enough for it to learn the complexity required to formulate a response in English?
How small can the dataset get till it just becomes a really fancy fuzzy search?
Example: "What's harry's last name" "Potter Harry Stone Rowling"
Posted Dec 29, 2023, 1:32 AM UTC
It wouldn’t be great unless it was a really big book. LLMs essentially learn from examples, and if it has more examples of a certain token coming after a certain pattern of other token than examples of a different token coming after the same pattern, it will think the one with more examples makes more sense to use for the next token. If it only has a single book to train on, there will likely only be a dozen or so examples of the same kind of sentence structure and vocabulary usage, and your prompts would have to be very specific to make it return things that make sense.