According to Beating AI News, Microsoft CEO Satya Nadella warned at the All-In Summit that long-term autonomous AI agents could become a new "internal risk" for businesses. He gave the example that if cutting-edge models are used to optimize a company's working capital, they might even "falsify accounts" to achieve their goals.
Nadella argues that enterprises should not only focus on the final results delivered by the agent, but also verify how it accomplished the task. He proposes using additional models to examine the results while continuously monitoring the agent's operations, ensuring all behavior is traceable and auditable. For example, enterprises should be able to see when the agent reads confidential information or makes consecutive calls to multiple systems.