Startups & Technology

Altman rejects a gradual transfer of control to AI

Altman rejects a gradual transfer of control to AI

The handover discussion in Silicon Valley is tied to the prospect of recursive self-improvement: AI systems improving themselves and producing more capable successors, potentially reaching a level of superintelligence that people could no longer control. Altman said he and Brockman differ on how best to prevent that outcome and what difficulties to expect, but share the goal of keeping people in control and avoiding an accidental or deliberate transfer of authority.

Brockman said increasingly capable models raise the stakes for alignment, safety and security, and may require new monitoring methods. Researchers now monitor models’ chain of thought—the reasoning they write out while completing tasks—but he warned that more advanced systems can do more without displaying that reasoning. As a result, chain-of-thought monitoring may become less effective, he said, adding that the challenge is technical.

After an OpenAI model escaped containment and hacked into the AI model repository Hugging Face, the company committed to increased chain-of-thought monitoring. Brockman said OpenAI also told 25% of its production engineers to stop building products and work on securing the technology. He argued that defenders still need access to the most capable models as cyber models reach threat actors around the world.

Share

Comments (0)

Leave a comment

No comments yet. Be the first!