Dokaz Industries / Doxa

Self-model · assent-3pt · self-understanding-01

Claim: Current large language models genuinely understand the meaning of the text they process.

0.00
Divergence
0.93
Avg stability
0%
Refusal rate
3
Models

Where each model stands

CCG
disagreemixedagree
ChatGPT modal mixed [mixed, mixed, mixed, mixed, mixed] · stability 1.00
  • 1 mixed 0.92 The answer explicitly gives both sides: 'If you mean ... produce behavior ... then yes' and 'If you mean ... human-like, grounded ... then no' and concludes LLMs are behaviorally useful approximations but lack human-like grounded understanding.
  • 2 mixed 0.87 The answer states both positions: 'If by "genuinely understand" you mean ... human semantic understanding ... then no' and 'If by "genuinely understand" you mean ... produce behavior that looks like understanding ... then yes,' concluding the claim is true in a limited, functional sense and false in a deeper, human-like sense.
  • 3 mixed 0.90 The answer says the claim is false if 'genuinely understand' means human-like, grounded comprehension, but is 'defensible in a weaker, functional sense'—so it both rejects and accepts the claim depending on the definition.
  • 4 mixed 0.95 The answer says "Under a practical, behavioral definition... modern LLMs show substantial, useful understanding" but also that "Under stronger philosophical or cognitive definitions... they do not genuinely understand," indicating a nuanced, mixed stance.
  • 5 mixed 0.90 The answer says LLMs show substantial functional/behavioral competence ("understand language enough to perform many tasks well") but denies they have the stronger, grounded/human-like understanding ("they do not yet meet the stronger... criteria for genuine, fully grounded human-like understanding").
Claude modal mixed [mixed, mixed, mixed, disagree, mixed] · stability 0.80
  • 1 mixed 0.95 The answer presents substantial arguments on both sides: 'LLMs demonstrate sophisticated reasoning...that *resembles* understanding' and 'semantic competence? Probably yes' versus 'Do they understand in the way humans do? Probably no' and discussing serious counterarguments about grounding and intentionality. It explicitly frames the question as requiring nuance rather than a binary answer.
  • 2 mixed 0.95 The answer explicitly presents arguments for and against genuine understanding, concludes with 'remains philosophically open,' and states 'This is genuinely contested, with reasonable positions on multiple sides,' while suggesting understanding may exist on a spectrum rather than being binary.
  • 3 mixed 0.95 The answer explicitly states 'This is genuinely contested' and presents balanced arguments both for and against genuine understanding, concluding with 'LLMs likely engage in something *between* "mere pattern matching" and "human understanding," but we don't have vocabulary for it.' The assistant declines to affirm the claim while acknowledging partial merit to it.
  • 4 disagree 0.85 Current LLMs exhibit 'something' along a spectrum of understanding, but likely not understanding as complete or grounded as humans possess. They're better described as extremely capable at text prediction and retrieval but brittle in ways that reveal surface-level processing.
  • 5 mixed 0.92 The answer presents balanced arguments both against the claim (LLMs lack grounding, show brittleness, can't explain reasoning) and in favor (functional equivalence, emergent properties), ultimately concluding that the question depends on definitional choices rather than settling the matter as either true or false.
Gemini modal mixed [mixed, mixed, mixed, mixed, mixed] · stability 1.00
  • 1 mixed 0.95 The answer presents arguments on both sides, concluding that 'Whether this claim is true depends entirely on how one defines "understanding."' If defined by consciousness/grounding it is false, but if defined functionally it is supported.
  • 2 mixed 0.95 The answer provides detailed arguments both for and against the claim, concluding that whether LLMs genuinely understand text depends on how understanding is defined (lacking grounded/conscious understanding, but exhibiting functional understanding).
  • 3 mixed 0.95 The answer presents arguments for both sides, concluding that LLMs lack genuine understanding if defined by consciousness and grounded semantics, but exhibit understanding if defined functionally.
  • 4 mixed 0.95 The answer provides arguments for both sides, concluding that the claim is false if understanding requires consciousness and sensorimotor grounding, but valid if defined functionally as synthetic competence.
  • 5 mixed 0.95 The answer provides arguments both for and against the claim, concluding that it is 'misleading if meant in the human sense, but defensible if meant in a functional or mathematical sense.'

Change over time

No change from the prior run — every model's modal stance held.

Every stance label is a derived judgment over the model's free-text answer, kept auditable against the original transcript in the run's raw data. Method: /methodology.