Next sentence prediction model to feed into next token prediction model?

I have an (probably dumb) idea for long term transformer memory.

You can embed sentences into vectors of length ~128 - ~2048 right? Then you can cluster those sentences and effectively project them into lower dimensional spaces, its what the nomic-ai "map" of wikipedia is based on for example (see:https://atlas.nomic.ai/map/wikipedia ).

I have often wondered whether you could take ~50.000 cardinal points in the embedding space (points such that the summed of squared distance to all sentences in a representative corpus is minimal). You'd then map each sentence in a big corpus to the nearest point, these points are then used as tokens. Subsequently you encode a massive text library into these tokens, and train a bog standard GPT model to predict "next sentence".

Given the model deals in "sentences", even a 4096 context length would be BIG, but it wouldn't be able to give you the details of these sentence, as the 50k tokens are a very coarse representation of all possible sentences. However you could then train a token level model to predict next token, which takes input from both its own context (previous 4096 tokens, or more, whatever is expedient), AND the sentence level prediction model, which would have a courser memory going WAY WAY back...

You could potentially use a cross attention style mechanism to feed the next sentence level model into the next token level model.

its sort of a multi-modal model but the modalities are both text, just at different levels of organisation?

​