Legal technology experts Michael C. Maschke, Sharon D. Nelson, and John W. Simek warn on Mar. 31 that the growing use of autonomous artificial intelligence (AI) agents in law firms could introduce new risks beyond the well-known issue of AI-generated errors.
The authors say that while many lawyers are still focused on problems like inaccurate citations and fabricated cases produced by AI, the next wave of risk involves AI systems that can act independently with limited oversight. According to a recent study led by researchers at MIT, many widely used AI agent systems lack basic monitoring tools, meaningful transparency, and reliable stop controls. “In plain English, they can operate with limited oversight and, in some cases, with limited ability to shut them down cleanly,” the article says.
Unlike chatbots that only suggest text or draft documents, these agent-based systems can perform multi-step tasks such as sending emails or updating records without direct human supervision. The authors point out that this efficiency comes with increased risk if there is no strong governance structure in place: “An AI agent that functions outside meaningful human oversight is not merely a technological issue but also a supervision concern.” They add that legal professionals are required to supervise non-lawyers and maintain competence with their tools under existing professional rules.
The article highlights three main vulnerabilities identified by researchers: insufficient logging and monitoring capabilities; inadequate disclosure when an action is performed by an AI rather than a human; and missing or ineffective kill switches to halt problematic actions quickly. These issues could lead to significant consequences for law firms if mistakes go unnoticed or cannot be traced back for correction.
To address these concerns, Maschke, Nelson, and Simek recommend every firm considering or using agentic AI ask three questions: Can you clearly identify what the agent did? Where are the human checkpoints? Is there a reliable way to stop the system? They stress that detailed audit trails and true kill switches are essential safeguards for any deployment of autonomous agents.
The authors conclude by noting past experiences adapting to new technologies such as cloud computing have shown how important proper governance is: “AI agents are not inherently out of control. Without deliberate oversight, they can be.”
