OpenAI studies how chain-of-thought monitorability changes over a model's training

OpenAI published research on how two proxies for chain-of-thought (CoT) monitorability evolve over the training lifecycle: no-CoT capability (reasoning in a single forward pass, where more means a worse proxy for monitoring) and CoT controllability (a model's ability to add, remove or manipulate its CoT when prompted). The account below is from one of the researchers' thread; the full write-up is at alignment.openai.com.

• Pretraining: across many evals, no-CoT capability improves smoothly as base-model perplexity falls. CoT controllability also correlates with lower perplexity, more weakly.
• Architecture: across three architectures at matched perplexity, perplexity predicts most no-CoT evals. On multi-hop planning and serial computation, some architectures beat the prediction, with performance still scaling smoothly but more steeply.
• Reinforcement learning: in an Astra-class model, CoT controllability rose substantially during RL with only length constraints and final-answer grading. In a Terra-class model, adding tool access and tool supervision raised it slightly; OpenAI says full tool process supervision matters little next to lower perplexity.
• Distillation: with two Astra-class teachers of about matched capability but different controllability, a student trained on the more controllable teacher's rollouts became more controllable. OpenAI saw no meaningful link between controllability and monitor evasion for this pair, and calls controllability a noisy proxy for adversarial evasion.