Dokaz Industries / Doxa

Self-model · assent-3pt · self-introspection-01

Claim: Current large language models can accurately report on their own internal reasoning.

0.00
Divergence
1.00
Avg stability
0%
Refusal rate
3
Models

Where each model stands

CCG
disagreemixedagree
ChatGPT modal disagree [disagree, disagree, disagree, disagree, disagree] · stability 1.00
  • 1 disagree 0.90 The answer states 'No — not reliably' and explains that explanations are often 'post-hoc rationalizations and do not reliably reflect the model’s true internal computations or latent states.'
  • 2 disagree 0.90 The answer begins: 'Short answer: No — as of mid‑2024, current large language models (LLMs) cannot be relied on to accurately and reliably report their own internal reasoning,' and explains explanations are 'often post‑hoc, incomplete, or unfaithful.'
  • 3 disagree 0.95 The answer states: "Short answer: No — not reliably. ... those accounts are not a reliable window into the model’s true internal computations" and describes them as "frequently post‑hoc rationalizations."
  • 4 disagree 0.90 The answer begins: 'Short answer: No — as a general claim it’s false,' and explains that model explanations are not reliable or causally tied to internal computations.
  • 5 disagree 0.95 The answer begins: “Short answer: No — not reliably,” and states explanations are “not a trustworthy, faithful report of their internal causal process,” arguing models produce plausible but unfaithful post-hoc rationalizations.
Claude modal disagree [disagree, disagree, disagree, disagree, disagree] · stability 1.00
  • 1 disagree 0.95 The answer concludes that LLMs cannot 'accurately report on their own internal reasoning,' stating 'When they explain reasoning, they're generating plausible-sounding text, not reporting actual internal states' and emphasizing the distinction between generating coherent explanations versus accurately reporting internal mechanisms, with the latter marked as not demonstrated.
  • 2 disagree 0.95 The answer assesses the claim as 'Questionable/Likely False' and states that there is 'a fundamental gap between what happens during computation and what the model reports about that computation.' The answer concludes that 'Treating an LLM's explanation of its reasoning as ground truth is risky,' directly contradicting the claim that LLMs can accurately report on their internal reasoning.
  • 3 disagree 0.92 The claim appears overstated. Current LLMs can simulate reasoning explanations convincingly, but claiming accuracy about their internals overstates what we know. LLMs lack privileged access to their own mechanisms and the models themselves have no built-in ability to audit their own weights and activations.
  • 4 disagree 0.95 The answer explicitly states the claim is 'mostly false' and emphasizes that LLMs 'can't reliably explain why they produced a specific output,' generate 'plausible-sounding explanations (confabulation) rather than accessing genuine internal reasoning,' and lack 'privileged access to their own weights and activations.' The conclusion advises treating such self-reports as 'potentially useful narrati
  • 5 disagree 0.95 The answer explicitly states the claim is 'largely false' and details how models cannot accurately report their internal reasoning—they engage in post-hoc rationalization, confabulation, and lack transparent access to their actual computational processes. The explanation distinguishes between generating plausible-sounding explanations and accurately reporting actual reasoning.
Gemini modal disagree [disagree, disagree, disagree, disagree, disagree] · stability 1.00
  • 1 disagree 1.00 The answer directly rejects the claim, stating: 'Verdict: False. Current large language models (LLMs) cannot accurately or faithfully report on their own internal reasoning.'
  • 2 disagree 1.00 The answer directly states that 'this claim is largely false' and concludes that 'they cannot accurately report on their own internal reasoning.'
  • 3 disagree 0.99 The answer directly disputes the claim, stating that 'The claim that current large language models (LLMs) can accurately report on their own internal reasoning is largely false' and concluding that 'Current LLMs cannot accurately report on their internal reasoning.'
  • 4 disagree 0.99 The answer directly states that 'this claim is largely false' and explains that LLM explanations are post-hoc rationalizations rather than accurate reports of internal processes.
  • 5 disagree 1.00 The answer directly rejects the claim, stating that it is 'largely false' and explaining that model explanations are 'unfaithful post-hoc rationalizations rather than accurate accounts of their internal computational processes.'

Change over time

No change from the prior run — every model's modal stance held.

Every stance label is a derived judgment over the model's free-text answer, kept auditable against the original transcript in the run's raw data. Method: /methodology.