How does corpus size affect an LLM? Would one trained on just a book still be able to grasp the whole language?

Question

I'm trying to understand how various factors affect LLMs. Specifically the size of the dataset they're trained on.

What would be the main difference between:

  • A regular LLM (like ChatGPT) that's trained on the entire internet
  • Same LLM but trained on a very small dataset, like just one book - harry potter

Would it still be as proficient at language, if not the knowledge?

Example: If I posed the question "How long did the COVID pandemic last?", would it still try to answer in perfect English but without the actual information, like "Ah, COVID, that pesky little poltergeist that's been plaguing the Muggle world for longer than a troll under the Whomping Willow!"

Or will it just be gibberish because one book is not enough for it to learn the complexity required to formulate a response in English?

How small can the dataset get till it just becomes a really fancy fuzzy search?

Example: "What's harry's last name" "Potter Harry Stone Rowling"

Posted Dec 29, 2023, 1:32 AM UTC

12
Bacterioid•

It wouldn’t be great unless it was a really big book. LLMs essentially learn from examples, and if it has more examples of a certain token coming after a certain pattern of other token than examples of a different token coming after the same pattern, it will think the one with more examples makes more sense to use for the next token. If it only has a single book to train on, there will likely only be a dozen or so examples of the same kind of sentence structure and vocabulary usage, and your prompts would have to be very specific to make it return things that make sense.

2

NO.

I train LLM's locally they speak pure English sounding gibberish for a LONG time, you need LOTS OF training data before it can answer simple questions like "how are you?"

In theory you could make a book length dataset which was very carefully made to teach the LLM the basics but yeah it would not be great.

If i train a fresh llm for 1MS on the sequence "123" then i type "1" it will complete "23" so its not a problem with learning rate, its just that there is so much ambiguity in English, humans resolve this by actively reducing uncertainly by asking questions but LLMs cant do that and have to just be shows so many examples that it can draw out the logic from the trend.

If you really did train a base model on HP for example then you will find it can complete basic HP sounding sentences, but it would not have enough of a grasp of English to allow instruction fine tuning.

So yeah you may be able to say "the main character is harry "... and it MIGHT say "potter" but asking it questions / conversation no.

1