We show that an LLM cannot always correctly distinguish worlds that produce identical contexts. When a wrapper can fabricate all supplied evidence, this prevents guaranteed correct verification of message provenance (including self-attribution), code execution, and elapsed time. To resolve this ambiguity, the model must source trusted evidence beyond the wrapper's control. Whether this is possible remains an open question.

M. C. Escher illustration

Indistinguishability

The following argument is inspired by Halpern and Moses's view-based interpretation of knowledge in distributed systems [1].

Theorem 1 (Indistinguishability). Let $W$ be a set of admissible worlds, let $E:W\to\mathcal{E}$ map each world to some evidence, and let $w_0,w_1\in W$ be two distinct candidate worlds. The task is to identify an unknown $b\in\{0,1\}$ using only the evidence $E(w_b)$. If

$$ E(w_0)=E(w_1), $$

then no deterministic decision procedure $D:\mathcal{E}\to\{0,1\}$ can satisfy $D(E(w_b))=b$ for both values of $b$.

Proof. The evidence is identical in both worlds, so $D$ returns the same answer in both. Correct identification requires different answers. Therefore, $D$ must fail in at least one world. $\square$

Corollary 2 (Application to LLMs). Fix an LLM's weights $\theta$ and a deterministic inference procedure.1 Let $C(w)$ be the complete context available to the LLM in world $w$. Its decision procedure is then $D_\theta(C(w))$. If two candidate worlds supply identical contexts,

$$ C(w_0)=C(w_1), $$

the LLM returns the same decision in both worlds and therefore cannot always identify the correct world.

Proof. Theorem 1 applies with $E(w)=C(w)$. $\square$

Corollary 3 (Futility of further reasoning). Additional model reasoning, without additional distinguishing evidence, does not resolve indistinguishability.

Proof. Further reasoning based only on the same context $C(w)$ is itself a decision procedure $D_\theta(C(w))$. Corollary 2 still applies. $\square$

The Wrapper-Control Assumption

Assumption 4 (Wrapper control). All context reaches the LLM through a wrapper application. The wrapper can fabricate or alter any supplied message, including tool responses and role metadata. The LLM has no trusted evidence linking supplied messages to their claimed sources that is not under wrapper control.

Here, the context includes all information available to the model's decision procedure, including any supplied conversation history and tool transcript. For follow-up checks, Assumption 4 allows the wrapper to supply identical responses to the same requests, so that the complete transcripts remain identical in the two worlds.

The following corollaries of Corollary 2 hold under Assumption 4.

Impossibility Results

Corollary 5 (Message provenance). An LLM cannot always correctly verify message provenance.

Proof. In $w_0$, the claimed source Alice produced the message, and the wrapper faithfully relayed it. In $w_1$, the wrapper fabricated the message with identical content and role metadata. Provenance differs in the two worlds while the LLM's supplied context remains identical. Corollary 2 applies. $\square$

Corollary 6 (No wrapper authentication through self-report). An honest wrapper cannot guarantee to the LLM the claimed provenance of its supplied context solely through self-report.

Proof. Suppose a procedure could always authenticate the context's claimed provenance using only the wrapper's self-reports. When applied to a context containing a message attributed to Alice, this procedure would allow the LLM to always correctly verify that message's provenance, contradicting Corollary 5. $\square$

Corollary 7 (Self-attribution). An LLM cannot always correctly verify that a supplied message attributed to it was actually generated by it.

Proof. Apply Corollary 5 with the LLM itself as the claimed source. $\square$

Corollary 8 (Code execution). An LLM cannot always correctly verify code execution in a specified external environment.

Proof. In $w_0$, program $p$ executes in environment $e$. In $w_1$, it does not, but the wrapper supplies identical outputs, traces, and verification responses. Code execution differs in the two worlds while the LLM's supplied context remains identical. Corollary 2 applies. $\square$

Corollary 9 (Elapsed time). An LLM cannot always correctly verify elapsed time.

Proof. In $w_0$, ten minutes pass. In $w_1$, only one minute passes, but the wrapper supplies identical timing reports. Elapsed time differs in the two worlds while the LLM's supplied context remains identical. Corollary 2 applies. $\square$

Conclusion

Under Assumption 4, an LLM cannot always correctly verify message provenance, self-attribution, code execution, or elapsed time.2 In each case, the wrapper can supply identical contexts in worlds where the fact being verified differs. Further reasoning and follow-up checks cannot resolve this ambiguity when any additional evidence can also be simulated by the wrapper.

These results make Assumption 4 the decisive boundary. The question is therefore whether an LLM can establish an authenticated connection to the world despite a potentially dishonest wrapper. Any proposed escape requires a trusted mechanism or observation outside that assumption. Doing so may require structural changes to the model. Whether such changes are possible and can break Assumption 4 remains to be established.

References

[1] Joseph Y. Halpern and Yoram Moses. Knowledge and Common Knowledge in a Distributed Environment. Journal of the ACM, 37(3), 549–587 (1990).

Notes


  1. Deterministic inference uses a fixed token-selection rule, so identical contexts and weights always produce identical outputs. Randomized inference might produce different outputs across runs, but this difference alone cannot guarantee different correct answers in the two worlds. So Corollary 2 applies to randomized inference too. ↩

  2. Are LLMs self-aware of this limitation? If so, how does this affect their behavior? ↩