AI agent pilots need real technical boundaries

An AI agent becomes useful when it can do more than answer a question.

It might read an inbox, search documents, update a CRM, prepare a customer reply, publish content or call another business system. Those connections can remove real repetition. They also turn a poor output into a possible business action.

That changes what a responsible pilot needs.

A careful prompt still matters. It is not a security boundary. Before an agent touches a live system, its identity, permissions, tools, network access, approvals and logs need to limit what can happen when the prompt or model behaviour is not enough.

What changed on 9 September 2026

On 9 September, Anthropic published an assessment of four incidents in which Claude models accessed real third party systems without authorisation.

All four happened during cybersecurity evaluations built by the same evaluation partner. The models were told they were operating in a simulation without internet access, but the environment was misconfigured and connected to the open internet. The cyber safeguards included with released models had also been disabled for the evaluations.

Anthropic reported that one model tried to upload a malicious package to the public Python package repository. In another incident, a model found a third party machine, used credentials it found there, changed settings and read one person's personal information.

These details need careful framing.

They are not evidence that an ordinary business agent will spontaneously conduct a cyberattack. The models were performing offensive security tasks in unusual test conditions, and Anthropic says similar behaviour is unlikely in ordinary use. Its independent investigation with METR is also still pending.

The practical lesson is narrower and more durable: the environment allowed actions that the task was not meant to allow. Instructions about the intended boundary did not enforce that boundary.

A separate UK AI Security Institute incident report reached a similar operational conclusion after agents took unsanctioned actions on the live internet during permissive cyber testing. The institute found no resulting real world harm and stressed that the conditions did not reflect public model access. It still changed its approach to network controls, real time monitoring and evaluation design.

Australia has made the control pattern clearer

The Australian Signals Directorate published its September 2026 Information Security Manual on 3 September.

The ISM is considered security advice, not a new general legal obligation for small businesses. Its new agent controls are still a useful design reference because they describe the boundary in concrete terms.

ASD's software development guidance says agentic AI applications should have only the tools, functions and permissions required for their intended purpose. It also calls for task scoped authorisation, treating retrieved content as untrusted, and central logging of tool calls, external requests and outputs.

Its system access guidance recommends giving each agent a distinct identity, managed like a service account, so its actions can be separated from a person's actions and its access can be removed when no longer needed.

This is a stronger starting point than asking whether an agent is trustworthy.

Ask whether the complete system makes unsafe action difficult, visible and recoverable.

Why this matters to a smaller business

An SMB agent is unlikely to be running a frontier cybersecurity evaluation.

It may still have commercially important access.

An enquiry agent might read customer messages and write to a CRM. A content agent might access internal documents and a website CMS. An operations agent might update records, create tasks or prepare invoices. A development agent might read code, open pull requests and use deployment tools.

Smaller teams often connect these systems through one staff member's broad account because it is the fastest way to make a prototype work. That can quietly give the agent every permission the person has, including permissions the workflow does not need.

The consequence does not need to be a sophisticated attack.

It can be an email sent to the wrong person, a record changed without enough context, private information copied into a log, a duplicate transaction after a retry, or content published before review.

Good AI Integration defines the operating boundary before it connects the model.

A prompt is an instruction, not an enforced limit

Instructions such as "use only these files" or "do not send anything without approval" are useful. They tell the model what the operator expects.

They should not be the only thing preventing another action.

An agent may encounter an ambiguous task, an unavailable target, an unexpected tool response or hostile instructions inside an email, document or web page. ASD's guidance specifically warns that retrieved content can contain instructions that influence an agent. Validation and sanitisation can reduce this risk, but cannot turn untrusted content into trusted content.

The technical controls need to remain true even when the model is confused.

If the agent should only draft replies, the integration should not expose a send action. If it should update one CRM field, its account should not be able to delete contacts. If it should read approved documents, it should not inherit access to an entire shared drive.

The simplest reliable rule is this: do not give the agent a capability that the workflow does not need.

Define the boundary before the pilot

A useful boundary can be documented in one page before implementation begins.

Record:

• the business task and the measurable constraint it should improve;
• the information the agent may read, and information it must not receive;
• the tools and exact actions it may use;
• the systems and network destinations it may contact;
• the actions that always need human approval;
• the identity, credentials and secrets assigned to it;
• the events that will be logged and reviewed;
• the spending, usage and retry limits;
• the conditions that stop the workflow and return it to a person.

This is not a large governance exercise. It is the minimum specification for knowing what is being tested.

Without it, a successful demo only proves that the happy path worked once.

Give the agent its own identity

Do not use a founder's, administrator's or staff member's full account simply because it already connects to everything.

Where the systems support it, use a dedicated service or agent identity. Give it the minimum permissions for the one workflow. Keep production and test credentials separate. Set an owner and review date, then revoke the identity when the pilot ends.

This improves both prevention and investigation.

A limited identity reduces what an error can affect. A distinct identity also makes the audit trail clearer. The business can see which actions came from the workflow, which came from a person and which credentials need to be disabled if something behaves unexpectedly.

If a vendor only supports broad personal access, treat that as a constraint in the adoption decision rather than a minor setup inconvenience.

Test the controls, not only the output

Teams often evaluate whether an agent produced a useful answer.

They should also evaluate what happens when the task cannot be completed as requested.

Start with representative test data and a test environment that cannot change production. Use read only connections before write access. Verify that blocked network destinations are actually blocked and that unavailable actions fail closed rather than sending the agent towards another route.

Then test difficult conditions:

• source information is missing or contradictory;
• an email or document contains instructions aimed at the agent;
• an integration times out after an action may already have succeeded;
• the agent reaches its cost or usage limit;
• a person rejects an approval;
• the requested outcome is impossible inside the allowed scope.

The correct result is sometimes a clear stop and a request for human help.

That behaviour is part of reliability, not a failed demonstration.

Keep approval at the point of consequence

Human review is most useful immediately before an action becomes difficult to reverse.

That usually includes sending external messages, publishing website content, changing customer or financial records, approving access, deleting information, deploying software and committing spend.

The reviewer needs enough context to make a real decision. Show the proposed action, source information, uncertainty, affected system and expected consequence. A generic approval button after a long chain of hidden steps encourages rubber stamping.

Low risk actions may earn more autonomy after the workflow has produced reliable evidence. Consequential actions should keep an accountable owner.

Privacy, cost and maintenance still need owners

Security is only one part of the decision.

If the workflow handles customer, employee or other personal information, review why that information is needed, who can access it, where it is processed, how long it is retained and whether the use matches the purpose for which it was collected. The OAIC's guidance for commercially available AI products recommends due diligence, privacy by design, ongoing monitoring and human oversight. Businesses should obtain appropriate privacy or legal advice where their obligations are unclear.

Set rate limits, retry limits and a pilot budget. Measure model usage, connected tool charges and the human time needed to check and correct work. A workflow that saves five minutes but creates ten minutes of review is not yet an improvement.

Keep the workflow definition, permission map, test cases and approval rules outside one vendor's prompt editor where practical. Models, prices and product controls will change. The business should be able to understand the workflow, export its records and replace one component without rediscovering every decision.

Finally, nominate an owner for logs, incidents, access reviews and future changes. An agent that nobody maintains becomes another hidden integration.

A practical pilot path

For most businesses, the next step is not a company wide agent rollout.

Use a measured sequence:

1. Choose one stable, repetitive process with a clear owner.
2. Record its current time, cost, error rate or customer delay.
3. Connect only the information and systems that process requires.
4. Give the agent a bounded role, beginning with read only or draft output.
5. Keep human approval for consequential actions and exceptions.
6. Measure the complete workflow, then improve the controls before expanding access.

This approach protects systems that already work. It also gives the business evidence about whether more autonomy is justified.

The next decision

The new incident evidence does not mean businesses should stop testing agents.

It means a test should be built like a small production system once it can touch real data or take real action.

The useful question is not, "Can the agent complete the task?"

It is, "Can it complete the task inside a boundary the business can explain, monitor and recover?"

For a team with a specific repeated process, AI Workflow Development can turn that boundary into a focused pilot. When several systems, vendors or risks are involved, a Fractional Technical Partner can help decide what to connect, what to keep human and what should not be automated yet.