During the first years of generative AI, we became accustomed to a relatively simple interaction. A person makes a request. The model processes the information. The system generates a response. It may draft a document, summarise information, analyse data or propose code. But, broadly speaking, the action that follows still depends on a person.
AI agents change this relationship. An agent can receive an objective, determine the steps required to achieve it, use different tools, retrieve information, interact with enterprise systems and perform actions in pursuit of that objective.
AI is no longer limited to telling us what we could do. It begins to do it. And that change has important consequences for the way we design systems.
From generating content to executing actions
Imagine an AI assistant that we ask about the status of several orders. The system retrieves the available information and generates a summary. A person reviews it and decides what to do.
Now imagine an agent. It receives the objective of resolving delayed orders. It queries the ERP, identifies incidents, checks inventory, requests information from another system, prioritises cases, prepares customer communications and updates certain records.
The difference is not necessarily that the model is more intelligent. The difference is that it has tools and permissions that allow it to intervene in its environment.
This is one of the essential characteristics of agentic systems. A model can reason. An agent combines that capability with mechanisms that allow it to observe, decide and act.
An agent is more than a model
Treating agents as simply a new generation of models can lead to poor design decisions. The model is only one component.
An agentic system may include memory, tools, data sources, APIs, rules, planning mechanisms, identity systems, permissions, validation and observability components. It may also interact with other agents.
Its behaviour therefore depends not only on the quality of the model but on the entire execution environment we build around it.
- What information can it access?
- Which tools can it use?
- What actions can it perform?
- With which credentials?
- For how long can it maintain an active objective?
- When must it stop?
- At what point does it require authorisation?
These questions are just as important as selecting the model the agent will use.
Autonomy is not binary
Agents are sometimes discussed as though there were only two possibilities: autonomous systems or systems controlled by people. In reality, there is a wide space between the two.
An agent may simply collect information and propose an action. It may prepare the action and wait for approval. It may automatically execute certain low-risk operations while requesting authorisation for others. Or it may operate autonomously within clearly established boundaries.
Autonomy can be designed. And it should be designed according to risk, the reversibility of actions and context.
Allowing an agent to classify documents automatically is not the same as authorising it to make a payment, modify critical configuration or send external communications on behalf of an organisation.
The right question is not simply ‘Can it do this?’ but ‘How far do we want to allow it to go without intervention?’
Permissions become a central issue
When AI only generates text, controlling what information it receives is already important. When it can also act on other systems, permission management becomes critical.
An agent should not automatically inherit all the permissions of the user or system running it. It should follow the same principle used elsewhere in security: least privilege.
- Access only the data it needs.
- Use only the tools it requires.
- Perform only authorised actions.
- And do so only for as long as necessary.
This means integrating agent design with identity and access systems, corporate policies and authorisation mechanisms. An agent’s security should not depend solely on telling it through an instruction what it must not do.
Important restrictions should also exist outside the model.
Instructions are not security controls
This distinction is fundamental. Models respond to instructions. We can tell them not to perform particular actions, to request confirmation or to follow certain rules. But an instruction is not necessarily equivalent to a technical control.
A robust system needs to assume that the model may misinterpret a request, receive manipulated information or produce an unexpected decision. Certain restrictions should therefore be implemented within the architecture.
If an agent must not perform transactions above a particular amount, the system executing the transaction can enforce that limit regardless of what the model requests.
If an action requires human approval, the architecture can prevent its execution until approval has been received.
If a tool should only be available under certain circumstances, permissions can restrict access to it.
Intelligence may be probabilistic. Critical boundaries should not be.
When data also contains instructions
Agents introduce another particular challenge. To perform their work, they need to observe external information: documents, emails, web pages, databases, messages or outputs generated by other tools. But that information may contain instructions that the system incorrectly interprets as part of its objective.
This type of problem, associated with techniques such as prompt injection, makes it essential to distinguish between trusted instructions and data that merely needs to be processed.
The distinction matters. An email received by an agent may contain information it needs to analyse, but it should not be able to redefine the rules under which the agent operates.
Designing this separation between data, instructions and permissions will be one of the fundamental security challenges for agentic systems.
Knowing what the agent did
Autonomy also creates more demanding traceability requirements. For a chatbot, recording a conversation may be sufficient. With an agent, we need to understand a sequence.
- What objective did it receive?
- What information did it access?
- What intermediate decisions did it make?
- Which tools did it use?
- What actions did it attempt?
- Which were authorised?
- Which failed?
- What result did it ultimately produce?
This information is necessary to investigate errors, improve the system and establish accountability.
It also allows us to evaluate something particularly important: not only whether the agent achieved the expected outcome, but how it got there.
Two agents may apparently produce the same result through very different routes. In enterprise systems, the route matters too.
Designing for failure
Any system can fail. With agents, the issue takes on an additional dimension because an error can propagate through several actions.
Incorrect data can produce a wrong decision. That decision may activate a tool. The tool may modify another system, and that modification may become the input for a subsequent action.
The design therefore needs to consider from the outset what happens when something goes wrong.
- Can the agent be stopped?
- Can an action be reversed?
- Are there execution limits?
- Can the system detect anomalous behaviour?
- Are there operations that always require confirmation?
Recovery should not be added after proving that an agent works. It should be part of its architecture.
From human in the loop to human on the loop
As autonomy increases, the role of people changes too. Requiring human approval for every action will not always be efficient.
In some environments, a different model may be more appropriate: allow the system to operate within established boundaries while a person retains the ability to observe and intervene.
We move from human in the loop to human on the loop. The distinction matters. Rather than necessarily validating each individual step, the professional supervises a system that has been granted a defined degree of autonomy.
This requires effective interfaces, appropriate alerts and a clear representation of what the agent is doing. Human oversight remains.
But it evolves from execution towards control.
The real question about agents
Agents create significant opportunities for automating complex processes. They can connect knowledge, reasoning and action in ways that previous systems struggled to achieve. But precisely for that reason, they require a different architecture.
The more a system can do, the more important it becomes to determine what it can access, what it can decide and what it can execute. The objective is not to reduce its capabilities. It is to make those capabilities usable.
Because the real challenge with AI agents will not be making them capable of acting.
It will be ensuring that they can do so with enough autonomy to create value and enough boundaries to maintain control.