The paper that opened the model
It started with a paper.
Wes Gurnee and Max Tegmark had gone looking for space and time inside language models. Their experiments suggested that spatial and temporal coordinates could be recovered from internal activations with simple linear probes.
Places and dates were not merely facts the model could repeat. Some trace of their relationships appeared to be arranged inside it.
That possibility was enough. I downloaded GPT-2 Small to see what was in there.
There was no master plan for Mesocosm. No archive, no cartography, no impossible library. There was a model small enough to run locally and a question compelling enough to keep following: if information inside a language model had shape, what would happen if I tried to look at it?
Years later, I am still messing with the same model.
Everything else—the neurons, parcels, routes, rooms, atlases, field notes, and the archive itself—grew from that sustained encounter. The paper did not contain Mesocosm. It made opening the model feel worthwhile.
Neuroscience
cosplay
This is not the beginning. It is simply one of the earliest fossils currently recovered: evidence that activations had become something to play with, and that the play was beginning to acquire a language of its own.
What they
found
Gurnee and Tegmark tested representations of geography and time in Llama-2 models across several spatial and temporal datasets. Linear probes could recover real-world coordinates from internal activations, and individual neurons correlated with spatial or temporal variables.
This does not prove that a language model contains a complete world model, or that its internal organisation is straightforwardly linear. For this history, the important fact is more personal: the result made representation space feel like a place that could be investigated.
