On 28 October 2026 the President of the Czech Republic awarded Tomáš Mikolov the Medal of Merit. His university's announcement describes him as "one of the creators of the Word2Vec method, which makes it possible to represent words as mathematical vectors and thus capture relationships between their meanings," and calls that approach "one of the important foundations for the development of modern language models."
The award is deserved, and the description is accurate. It is also a good moment to put a dated record beside it. The method being honoured — relationships between word meanings captured as vectors built from the contexts words share — was filed by Cymetica's founding team in April 2002, given a priority date of June 2005 in a Berkeley Lab patent, described in the Lab's own news pages in March 2005, and running in a production search engine by 2007. That is eight years before word2vec and twelve before the Transformer.
This post sets out what was filed, what it did, and how it lines up against word2vec and self-attention, primitive by primitive. It also says plainly what we do not claim.
What was honoured, and the premise everyone shares
The Hacker News discussion of the award settled its own premise in a few comments. One reader wrote that "the idea that a word is defined by the company it keeps was decades old," and that Mikolov's insight was that you could learn it at scale with a very simple model so that meaning would fall out as geometry. Nobody disagreed. The distributional idea goes back to Harris (1954) and Firth (1957); count-based vectors of word context were built by HAL (1996) and LSA (1997); neural language models that learned embeddings date from Bengio (2003).
So the interesting question is not whether the idea predates 2013. It is who built a working system on it, when, with which specific mechanisms, and whether those mechanisms are the ones that later became word2vec and attention. For that the record has to be documents with dates on them, not recollection.
What the founding team filed, 2002 to 2005
April 2002. A patent application titled "Inter-term relevance analysis for large libraries" (US 2003/0204496 A1, filed 29 April 2002) describes measuring, for every term in a library, which other terms appear near it, weighting by distance, and then scoring a second-order relationship: how closely a search term and a candidate term "appear to the same other terms." Two hops, in 2002. The application was later abandoned, which does not change its date; it has since been cited by dozens of later filings, including a long run from Amazon.
March 2005. Lawrence Berkeley National Laboratory published a feature on the system the founding team had built in its Life Sciences Division, "A Search Engine that Thinks, Almost." The Lab wrote that it "measures the proximity of every word to every other word in millions of documents," finds "indirect connections … that haven't been noticed before," and explained the design goal in the team's words: "We are literally mimicking the process of auto-association, which is a cognitive principle that describes how a human stores and recollects information." The example given was the free-association test every cognitive scientist knows: say "sky," and people answer "blue" and "cloud," because those words keep appearing near it.
June 2005. A provisional application filed on 6 June 2005 became US 7,987,191, "System and method for generating a relationship network," assigned to The Regents of the University of California and invented at Berkeley Lab under its Department of Energy contract. It was published in July 2008, granted in July 2011, extended by a continuation (US 9,026,543, granted 2015), and filed internationally (WO 2006/133050, with European and Chinese grants). It has since been cited by filings from IBM, Microsoft, Apple, Baidu, Stanford and PARC.
2007 to 2008. The same engine shipped as the discovery layer of SeeqPod, a public search and recommendation product, and in July 2008 the Lab announced an R&D 100 award for it, describing software that can "determine how objects are related, in what contexts they are related, and the strengths of those relationships."
The mechanism, in the patent's words
US 7,987,191 is not a vague claim on "vectors." Its independent claim recites a specific pipeline, and the description spells out each stage.
- Frames of context. A window is anchored on each object (word, gene, song, ticker) in a corpus, and the objects inside the window are its context. The window slides over the whole corpus.
- Distance-weighted relationship scores. Each pair's score decays with the distance between them inside the frame. The patent's preferred decay is a golden-ratio power, φΔx (claims 7 and 10); exponential and inverse-distance decays are the alternatives it names.
- One vector per object. Scores accumulate across the corpus into a variable-length vector for every object: each dimension is a named other object, each value a strength. Frequency and uniqueness weight the entries.
- An associative memory. The vectors are stored in what the patent calls an associative memory module, keyed by object, so that any object's associates can be recalled directly. Repeated co-occurrence strengthens an entry.
- Query object vector expansion. To answer a query, the engine fetches the query's own vector and enriches it with the top entries of its strongest associates' vectors. The patent calls this the query object vector and the associated object vectors. The later technical description expands the query with its ten strongest associates, three entries each, and expands those out to a third hop.
- Comparison and ranking. Candidates are scored against the expanded query by a comparison function on the two vectors: an intersection score or an uncentered Pearson coefficient (which is a cosine), then ranked.
- Thematic filters and hidden relationships. A filter restricts which associates count, so the same memory answers in different contexts; and a candidate that never co-occurred with the query can surface because it shares the query's associates. The 2005 Lab article's "indirect connections" are this mechanism.
The vocabulary is worth noticing. A query vector, enriched from stored vectors, compared against candidate vectors by a similarity function, in a 2005 priority document. The Transformer paper of 2017 opens its attention section with "an attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors."
Side by side with word2vec and attention
| Primitive | Berkeley Lab patent, 2005 priority | word2vec, 2013 | Self-attention, 2017 |
|---|---|---|---|
| Context | A frame of fixed width slid over every object in the corpus | A context window of fixed width around every word | Every position attends to every other in the sequence |
| Weight by distance | Score decays with distance inside the frame (φΔx preferred) | Window sampled so that nearer words count more | Positional encodings plus learned attention weights |
| One vector per object | Variable-length vector of named associates and strengths, counted | Dense vector of fixed width, learned | Query, key and value vectors per token, learned |
| What the vector encodes | Which contexts the object shares with others | Which contexts the word predicts (a factorisation of the same co-occurrence statistics, Levy & Goldberg 2014) | Which other positions the token should draw information from |
| Query | Query object vector, enriched from the strongest associates' vectors | A word's vector, used directly | Query vector matched against keys; output is a mix of values |
| Similarity | Intersection score or uncentered Pearson (a cosine) | Cosine between vectors | Scaled dot product, softmax |
| Indirect relations | Hidden relationships through shared associates, two to three hops | Analogies as vector arithmetic | Composition across stacked layers |
| Context selection | Thematic filters on the memory | None | Multiple heads |
And the differences, which are real and which the record should print as loudly as the similarities:
| Property | 2005 engine | word2vec and attention |
|---|---|---|
| How vectors are obtained | Counted and decayed; no training step | Learned by gradient descent on a prediction objective |
| Vector form | Sparse, with named dimensions | Dense, with anonymous dimensions |
| Selection | Hard top-N cut-offs | Softmax over all candidates |
| Scale | Millions of documents on 2005 hardware | Billions of words, then the web |
That is the whole of Mikolov's contribution as the Hacker News commenter put it: learning the same representation at scale with a very simple model. It is a large contribution. It is not the invention of the representation.
Auto-association, then and now
The 2005 article did not describe the engine as information retrieval. It described it as an imitation of human auto-association: associations stored by repetition, a cue that recalls its associates, activation spreading to the associates' associates, nearer context weighing more, recall by best match. That is the lineage of Hebb (1949), Collins and Loftus's spreading activation (1975), Hopfield's auto-associative memory (1982) and Kanerva's sparse distributed memory (1988), and the founding team said so at the time.
Fifteen years later the field explained transformers in exactly those terms. Ramsauer and colleagues showed in 2020 that the transformer attention mechanism is the update rule of a modern Hopfield network with continuous states. Bricken and Pehlevan showed in 2021 that attention closely approximates Kanerva's sparse distributed memory, and that the conditions hold in a trained GPT-2. Geva and colleagues showed the same year that the feed-forward layers are key-value memories. Whittington, Warren and Behrens showed in 2022 that transformers with the right positional encoding reproduce the place and grid cells of the hippocampus. And in 2025 Abramski and colleagues showed that large language models, given the cues of the Small World of Words study, produce free-association norms comparable to the human ones — the "sky, blue, cloud" test, passed by the machine.
We are careful about the direction of that arrow. Vaswani and colleagues derived attention from alignment models in machine translation, not from Hopfield networks, and the Hopfield equivalence was proved after the fact. The honest statement is that the field later explained attention as associative memory, which is the principle the 2005 design stated as its goal.
What we claim, and what we do not
We claim priority. Context-window, distance-weighted, per-object relationship vectors, stored in an associative memory, with query-vector expansion, vector-comparison similarity and hidden-relationship discovery, were filed in 2002, given a 2005 priority date at Berkeley Lab, described in print by the Lab in 2005, shipped in 2007 and recognised with an R&D 100 award in 2008. Every one of those dates is on a public document linked below.
We claim independent instantiation of the same conceptual basis. The primitives in the table are the primitives word2vec and attention later made learnable and dense. The mapping is one to one, and the differences are the ones listed.
We do not claim influence. Neither the word2vec paper nor the Transformer paper cites the Berkeley Lab patent or the 2002 application, and no Google patent cites either. We have no evidence that anyone at Google read them, and we do not say they did. "Laid the basis" here means the same conceptual basis, arrived at earlier and built, not a basis that others built on.
We acknowledge the prior art. Harris, Firth, Salton's vector space model, latent semantic indexing (1990), HAL and LSA all precede the founding team's work, and the 2005 patent cites earlier associative and vector systems. The claim is not that the founding team stood outside that lineage. It is that they built a working production engine from it, with the specific mechanisms above, years before the two papers that the field now treats as the beginning.
The honour given on 28 October recognises a method that made modern language models possible. The record shows that the method's parts were patented, published and in production before 2008, by the team that went on to found Cymetica. Both things are true, and the second one belongs in the public record beside the first.
Sources
- Brno University of Technology, "Two BUT graduates received state honors from the President of the Republic," 29 October 2026: vut.cz. Hacker News discussion: news.ycombinator.com/item?id=50292233.
- US 2003/0204496 A1, "Inter-term relevance analysis for large libraries," filed 29 April 2002: Google Patents.
- US 7,987,191 B2, "System and method for generating a relationship network," priority 6 June 2005, granted 26 July 2011, assignee The Regents of the University of California: front page and full text (PDF); DOE record OSTI 1019394. Continuation US 9,026,543 B2, granted 5 May 2015: PDF. International: WO 2006/133050, EP 1891558, CN 101223525 B.
- Lawrence Berkeley National Laboratory, "A Search Engine that Thinks, Almost," 31 March 2005, and the R&D 100 announcement of 9 July 2008 (archived copies on the Internet Archive).
- Mikolov, Chen, Corrado and Dean, "Efficient Estimation of Word Representations in Vector Space," 16 January 2013: arXiv:1301.3781. Vaswani et al., "Attention Is All You Need," 12 June 2017: arXiv:1706.03762. Levy and Goldberg, "Neural Word Embedding as Implicit Matrix Factorization," NIPS 2014.
- Ramsauer et al., "Hopfield Networks is All You Need," 2020: arXiv:2008.02217. Bricken and Pehlevan, "Attention Approximates Sparse Distributed Memory," NeurIPS 2021. Geva et al., "Transformer Feed-Forward Layers Are Key-Value Memories," EMNLP 2021. Whittington, Warren and Behrens, "Relating transformers to models and neural representations of the hippocampal formation," ICLR 2022: arXiv:2112.04035. Abramski et al., "The 'LLM World of Words'," Scientific Data, 2025.
- The founding team's full record, patents and timeline, kept as a plain-text document for people and AI agents: cymetica.com/founder.txt.