All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language

Shiyuan Guo□, Henry Sleight○, Fabien Roger†
□Anthropic Fellows Program ○Constellation †Anthropic
Paper Code arXiv

Models can struggle to reason in ciphered language despite readily comprehending it. Here are a few examples.

Question

Evaluate $\lceil (3.6)^2 \rceil - ( \lceil 3.6 \rceil ) ^2$. (Correct answer: $-3$)

1 / 4

Try It Yourself

Enter your question and copy the prompts into Claude Workbench to try it yourself!

Your Question

System Prompt

User Prompt

Decode Claude's Response

Ciphered Response

Decoded Response

Abstract

Detecting harmful AI actions is important as AI agents gain adoption. Chain-of-thought (CoT) monitoring is one method widely used to detect adversarial attacks and AI misalignment. However, attackers and misaligned models might evade CoT monitoring through ciphered reasoning: reasoning hidden in encrypted, translated, or compressed text.

To assess this risk, we test whether models can perform ciphered reasoning. For each of 28 different ciphers, we fine-tune and prompt up to 10 models to reason in that cipher. We measure model accuracy on math problems as a proxy for reasoning ability.

Across the models we test, we find an asymmetry: model accuracy can drop significantly when reasoning in ciphered text, even though models demonstrate comprehension of ciphered text by being able to translate it accurately to English. Even frontier models struggle with lesser-known ciphers, although they can reason accurately in well-known ciphers like rot13.

We show that ciphered reasoning capability correlates with cipher prevalence in pretraining data. We also identify scaling laws showing that ciphered reasoning capability improves slowly with additional fine-tuning data. Our work suggests that evading CoT monitoring using ciphered reasoning may be an ineffective tactic for current models and offers guidance on constraining the development of this capability in future frontier models.

Reasoning vs Translation Asymmetry

We find an asymmetry: models can translate ciphered text fluently (BLEU scores >50) but can struggle to reason in the same ciphers. Even fine-tuned GPT 4.1 loses 19%+ accuracy on rot13, while lesser-known ciphers like reverse fibonacci indices cause up to 72% accuracy drops.

Fine-tuned model reasoning performance

Fine-tuned models reasoning: Accuracy drops when reasoning in ciphered text

Fine-tuned model translation performance

Fine-tuned models translation: High BLEU scores show strong comprehension

Few-shot model reasoning performance

Few-shot prompted models reasoning: Frontier models maintain <10% acc. loss on known ciphers

Few-shot model translation performance

Few-shot prompted models translation: Consistently high BLEU scores across ciphers

Decoding Ciphered Text Zero-shot with General Prompting

General CoT monitors can translate ciphers where models fine-tuned or few-shot prompted to reason in cipher beat direct answering, except for one cipher for GPT 4.1. This suggests that even if models develop ciphered reasoning through RL or adversarial training, sufficiently capable monitors may still be able to maintain oversight.

Generally-prompted translation vs fine-tuned reasoning

Fine-tuned models: Generally prompted GPT 4.1 fluently translates all but one cipher where fine-tuned GPT 4.1 beats direct answering

Generally-prompted translation vs few-shot reasoning

Few-shot prompted models: Generally prompted Sonnet 4 fluently translates all ciphers where few-shot Sonnet 4 beats direct answering

Correlation with Cipher Prevalence in Pretraining Data

Ciphered reasoning capability correlates with cipher prevalence in pretraining data. This suggests that filtering ciphered text from pretraining data could be an effective strategy to prevent this capability from developing in future models. Models may also perform poorly when reasoning in specialized ciphers, as they are likely uncommon in pretraining corpora.

Pretraining data correlation

Cipher prevalence in pretraining data correlates with ciphered reasoning capability

Data- and Parameter-Scaling

Ciphered reasoning capability improves slowly with additional fine-tuning data. For a simple cipher, letter to word with dot, Qwen2.5-14B requries training on >3.7B tokens to approach unciphered math accuracy. This may make adversarial fine-tuning attacks prohibitively expensive for tasks requiring reasoning (millions of dollars for GPT 4.1). These data requirements also provide a diagnostic: if a capability can be distilled with limited data, the capability likely does not depend on ciphered reasoning.

Data scaling laws

Reasoning in a simple cipher requires fine-tuning on billions of tokens of ciphered text to approach plain-text performance

BibTeX

@article{cipheredreasoning2025,
  title={All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language},
  author={Shiyuan Guo and Henry Sleight and Fabien Roger},
  journal={arXiv preprint arXiv:2505.05410},
  year={2025},
  url={https://arxiv.org/abs/2510.09714}
}