Do large language models understand meaning or just predict words?

Large language models are trained to predict words, and that much is not in dispute. Whether this process produces understanding of meaning is an open and heated debate among AI researchers. One camp holds that training on text alone cannot yield meaning. Another holds that the models develop internal representations with real semantic value, though not necessarily humanlike ones. The answer depends heavily on what one means by "meaning" and "understanding". [1] [2]

During training, these networks predict a hidden part of an input sentence, a method called self-supervised learning. The result is a complex statistical model of how words and phrases in the training data correlate. Linguist Ellie Pavlick, interviewed by Quanta Magazine, describes the goal as predicting the next word in a sequence. She notes that when such a model is asked to explain what it just said, it is not faithfully reporting a reason. It produces the kind of words that tend to follow such a question in its training data. [1] [3]

Emily Bender and Alexander Koller argued in 2020 that a system trained only on linguistic form has no way, in principle, to learn meaning. They define meaning as the relation between a linguistic form and the communicative intent it can be used to evoke. On this view, claims that models "understand" language are overclaims. They allow that such models may become useful components of a future system that is truly humanlike. [4]

Others on this side stress embodiment. Humans know what a tickle is because they have bodies, whereas a model can use the word without the sensation. A 2024 review draws on semiotics and says that defining a word with other words can be done from the relations among forms alone, without reference to what the words signify. Language is itself a model of meaning, so models trained on vast text can mimic meaning convincingly without having it. Skeptics also point to the Eliza effect, the human tendency to attribute understanding to machines that show even faint humanlike language, and they say surprise at the models' fluency reflects our weak intuition about what statistics can do at scale. [1] [5]

Researchers at Amazon propose defining a sentence's meaning by the distribution of continuations it makes likely. Sentences with the same distribution fall into one equivalence class. They report that large models develop internal representations with semantic value, composed of discrete entities that relate through directionality, entailment and containment. They stress that this does not show models grasp meaning as humans do. They also argue that no one can directly see the meaning behind an expression, for a model or a human, and that shared data and experience give a basis for communication. [2]

Even the skeptical review concedes that work claiming meaning in models usually qualifies it as a representation or approximation of meaning, which still gives impressive functional performance. [5]

The search results do not show a settled consensus. Both sides agree that the models are fluent and that they encode something about meaning. They differ on whether that something counts as understanding, and on whether grounding in the physical world is required. The disagreement is partly about definitions. [1]