Meaning as geometry
How words, then every kind of object, became vectors — and how vector space became the core of language models.
- July 1945 Vannevar Bush, "As We May Think"
- 1954 Zellig Harris, "Distributional Structure"
- 1957 J. R. Firth: "the company it keeps"
- 1957 Osgood: the semantic differential
- 1958 Luhn: term significance by formula
- 1972 Kohonen: correlation matrix memories
- 1972 Spärck Jones: inverse document frequency
- 1975 Salton, Wong and Yang: the vector space model
- 1982 Hopfield networks
- 1986 Swanson: undiscovered public knowledge
- 1986 Backpropagation
- 1990 Latent semantic analysis
- 1990 Elman: finding structure in time
- 1992 Brown clusters
- 1996 HAL: hyperspace analogue to language
- 1997 Landauer and Dumais: Plato's problem
- 1997 LSTM
- 1998 Gordon and Dumais: LSI for discovery
- April 2002 Inter-term relevance analysis (US 2003/0204496 A1)
- 2003 Bengio et al.: neural probabilistic language model
- 2003 Latent Dirichlet allocation
- 31 March 2005 Berkeley Lab: "A Search Engine that Thinks, Almost"
- 6 June 2005 Relationship network patent filed (US 7,987,191)
- 2006–2009 SeeqPod: the engine on objects
- 2008 Collobert and Weston: one network, many tasks
- 9 July 2008 R&D 100 award: the Biomimetic Search Engine
- 2010 Turian et al.: word representations
- 26 July 2011 US 7,987,191 granted
- January 2013 word2vec
- September 2014 Bahdanau et al.: attention
- October 2014 Neural Turing machines and memory networks
- 2014 GloVe
- 2016 item2vec and node2vec
- 2016 fastText
- June 2017 "Attention Is All You Need"
- February 2018 ELMo: contextual word vectors
- June 2018 GPT
- October 2018 BERT
- November 2018 Context-controlled correlation matrices
- February 2019 GPT-2
- July 2019 Tshitoyan et al.: latent knowledge in embeddings
- August 2019 Sentence-BERT
- November 2019 kNN-LM
- May 2020 Retrieval-augmented generation
- May 2020 GPT-3
- July 2020 "Hopfield Networks is All You Need"
- February 2021 CLIP
- August 2021 ALiBi
- December 2021 RETRO
- March 2022 Memorizing Transformers
- September 2022 Toy models of superposition
- November 2022 ChatGPT
- July 2023 RetNet
- November 2023 The linear representation hypothesis
- May 2024 Scaling monosemanticity