How do you keep a person in the loop when an AI agent does real work?
Keep a person in the loop by deciding, action by action, whether the agent can act alone, needs approval or must hand over. Use clear rules rather than the model’s own confidence to trigger reviews, give every escalation a named owner, and log every action so you can see who decided what and when.
Human in the loop is a dial, not a switch
“Human in the loop” is often used as if an agent is either supervised or not. In practice, each action an agent can take sits somewhere on a scale, and different actions in the same agent sit at different points.
| Level | How it works |
|---|---|
| Suggest | The agent recommends; a person decides and acts. Useful for judgement calls such as which applicant to shortlist. |
| Draft for approval | The agent prepares the action, such as a reply, a refund or a post, and a person approves, edits or rejects it. |
| Act and notify | The agent acts, then tells a person what it did, so it can be checked and reversed if needed. |
| Act silently | The agent acts and logs it. Only for low-risk, reversible, high-volume actions. |
Decide the level per action when you scope the agent, and expect some to move over time as the logs show what is reliable.
How to design approvals people will actually do
An approval step that annoys people gets ignored or rubber-stamped. Good approvals are quick, come to where the approver already works and show everything needed to decide.
- Put approvals where people are. For a business owner who lives on their phone, approving in WhatsApp beats logging into a dashboard. Our Sosho agent lets a shop owner approve or redirect a week of social posts in a single message thread.
- Show the evidence. Include the request, the proposed action and why the agent chose it, so the approver is not reconstructing the case.
- Batch where you can. Ten similar approvals in one list are faster than ten interruptions.
- Decide what happens on silence. If nobody approves within a set time, the action should wait or go to a backup person, not proceed by default.
- Make editing easy. Approvers often want to tweak rather than reject. Capture the edits; they show where the agent needs adjusting.
Watch for rubber-stamping. If an approver accepts every item within seconds, the check has stopped working. Either the action is reliable enough to move to act-and-notify, or the approver is overloaded and needs fewer, better-presented requests. Approval logs that record how long each decision took make this visible.
Confidence thresholds: what should trigger a review?
It is tempting to ask the model how confident it is and send anything below a number to a person. Treat that with caution. A language model’s stated confidence is not a reliable probability; it can be confidently wrong. Use it, if at all, as one signal among several.
More dependable triggers are rules your software checks outside the model:
- Value limits: any refund, discount or payment above an amount you set.
- Novelty: a new customer, a new supplier or a request type the agent has not handled before.
- Mismatch: the extracted data does not match your records, such as an invoice total that differs from the purchase order.
- Missing information: a required field is empty or unreadable.
- Sensitive content: complaints, health information, legal threats or signs that someone is vulnerable.
- Disagreement: two independent checks, or two passes of the model, produce different answers.
Start with thresholds set cautiously, so more goes to review than strictly needed. Lower them as the review data shows the agent is reliable for a given case.
Design your approval points.
Human-in-the-loop design
0 of 9 done
Escalation: handing over without dropping the ball
An escalation that lands in a shared inbox nobody watches is a failure dressed as a safeguard. For each type of escalation, define:
- Who receives it, by name or role, with a backup.
- Which channel it arrives in, and how urgent it is.
- What summary the agent includes: the request, what it has done so far and why it stopped.
- What the customer is told while they wait, and when.
- How the person closes it, so the record shows the outcome.
For example, in a hypothetical property management company, a tenant reports a leak at night. The agent recognises it as urgent, sends the on-call contractor a WhatsApp message with the address, photos and access notes, tells the tenant who is coming and when, and records the job. If the contractor does not confirm within the agreed time, it calls the backup. The person still makes the repair; the agent removes the chasing.
The agent can still save time on escalated work by writing the summary and gathering the records. For customer-facing agents, see customer support for how this fits a support workflow.
Audit trails: record who decided what
Every action an agent takes should leave a record: the input it received, the tools it called, the action it took or proposed, who approved it, and when. Keep the version of the instructions and configuration in use at the time, so you can explain behaviour after changes.
This matters for fixing mistakes, for answering customers and for compliance. On the Smurfs FC platform, every discount needs the owner’s sign-off and each override is logged with who approved it, when and why. That same discipline applies to agents. Balance it against data protection: log what you need, for as long as you need it, as covered in GDPR and AI agents.
When is full automation fine?
Full automation is reasonable when an action is all of these: low in cost if wrong, easy to reverse, quickly noticed if it goes wrong, and checked by rules outside the model. Logging attendance, sending a standard reminder from an approved template or filing a document in the right folder often qualify.
Decisions with legal or similarly significant effects on people, such as refusing a service or an application, deserve particular care. GDPR places limits on solely automated decisions of that kind, so check with your advisor before removing the person. For the security side of what an agent can do on its own, read AI agent security, and see AI agent development for how we build these controls in.
Common questions
Does human review defeat the purpose of an AI agent?
No. Reviewing a prepared action takes far less time than doing the task from scratch, and the agent still gathers the information, drafts the result and records the outcome. Many agents save most of their value on the routine work while a person approves only the sensitive or unusual cases.
Who should approve an agent’s actions?
The person who would make that decision today, or someone with the same authority. Approving a refund should sit with whoever can approve refunds now. Avoid routing everything to one busy owner, which creates a bottleneck, and give each approval type a backup so work does not stall during holidays or sick days.
How do we know when to reduce human review?
Look at the approval logs. If approvers have accepted a type of action unchanged for a sustained period, with no complaints or corrections afterwards, it may be ready for act-and-notify. Move one action type at a time, keep monitoring, and be ready to move it back if errors appear.
Can a person override the agent after it has acted?
They should be able to. Design reversals for each action: cancel the booking, reverse the entry, send a correction. Record the override in the audit trail with the reason. Overrides are also valuable feedback, because they show where the agent’s rules or instructions need adjusting.
Bring us the task.
Tell us what happens today, which tools are involved and where it goes wrong. On a free 30-minute review, we’ll work out whether an agent, an automation or neither is the right fix, and what a sensible first release would be.
Start a project.

Book a call or send us a message. We read everything.
tash@neu.ie
Working with teams in Ireland and worldwide.
Free 30-minute AI review.