Early language models - how did they pull it off?

Do you remember Tay, the Microsoft chatbot from 2016? Or (earliest generation of) Xiaoice from 2014? Despite the fact that AI technology has been around for many years, I find it increasingly difficult to imagine how they managed to do it back then.

The paper 'Attention is All You Need' was published in 2017, and the GPT-2 paper ('Language Models are Unsupervised Multitask Learners') in 2019. Yes, I know we had RNNs before that could do a similar thing, but how on earth did they handle the training dataset? Not to mention their ability to learn from many conversations during inference, which is also what got Tay taken down after only a day.

I don't think they even used the design principle as modern LLMs. It's a shame that I can't find any official information about Tay's architecture, as well as how it's trained...