In reasoning model development, it's always been this interesting tension between training/optimizing chain-of-thoughts (CoTs) vs not touching them to preserve monitorability.

CoTs are cool because we can check what the model is up to (e.g., whether it cheats or not) as long as it spells out its thought process. And we risk losing that if we explicitly optimize CoTs.

Two caveats:

1) I.e., the catch of CoT and monitorability is that this only works if the written thinking reflects what the model is actually doing.

2) Also, when models become more capable, they spell out less in the CoT (like a cleverer math student needs less note-taking when solving problems during an exam).

Nothing new on that front, but an interesting thread below on how to measure "harder to monitor", with some interesting results that confirm the existing intuitions:

1) no-CoT capability (meaning how much reasoning the model can do in a single forward pass without writing anything down)

• Better pretraining (lower perplexity) raises this no-CoT capability
• Architecture mostly doesn't matter
• CoT controllability (meaning how well the model can change its written reasoning when asked to add, remove, or change parts of it)

• RL alone raises controllability a lot
• Controllability is passed on through distillation

* Higher no-CoT capability and higher controllability both make monitoring harder.

PS: Perhaps the good news is that it seems that process supervision wasn't the main culprit.