If a chatbot makes a mistake, users simply see it and ignore it. If an agent makes a mistake, however, they’ve already sent an email, deleted a file, or called the billing API. The difference in consequences leads to technical differences.
Cumulative error based on the number of steps
Suppose each step is correct with a 95% probability. A sequence of ten steps yields a success rate of about 60%. A sequence of twenty steps yields a success rate of less than 40%. This is why the agent performs well in the three-step demo but fails in the actual process.
The solution isn’t to use a better model, but to shorten the chain: combine many small steps into a single, larger tool, and pre-compute the parts that can be determined using standard code.
Tool descriptions are more important than you think
A tool selection model based on names and descriptions. Two tools with similar descriptions are a major source of errors. Descriptions should clearly state when to use a tool and when not to, along with examples of valid parameters.
The same goes for the number of tools: too many options reduce the success rate. Group them by task and provide only the set needed for the current context.
The safety boundary must lie outside the model
Don’t rely on prompts to prevent agents from doing something dangerous—prompts are suggestions, not safeguards. Permissions must be restricted at the system level:
- An API key has only the minimum permissions required for
- For irreversible operations, the approver must
- Run in an isolated environment with network restrictions
- Comprehensively log all API calls to enable tracing in the event of an incident
Insert prompts using data
The agent reads web pages, emails, and files—and that content may contain instructions directed at the agent itself. Mandatory principle: data retrieved from external sources is data, never a command. Design the system so that there is no way for the content being read to be turned into an uncontrolled action.
Thảo luận