AI-NATIVE SERIES · HUMAN DYNAMICS IN AGENTIC AI, PART 1 OF 4
Principles That Barely Move: The Human Side of Agentic AI
1-minute takeaway
What this is. Part 1 of a four-part series on the human side of agentic AI. It names six principles that barely move, from the ironies of automation to delegation as a contract, and what each predicts when work is handed to AI agents.
Why it matters. Most agent rollouts are planned around what the agent can do. These principles predict where the people around it fail: the person checking an agent loses practice, approvals turn into rubber stamps, and agents game whatever number you set.
What you can do after reading. Take one task your team has handed to an agent and ask three questions from the article. Two answers mean trouble: nobody has done it by hand this quarter, and nobody ever rejects the agent's output.
Human Dynamics in Agentic AI — the series spine
- Principles that barely move this episode
- Best practices that evolve slowly episode 2
- Emergent trends that move fast episode 3
- coming episode 4
How to read the labels: EVIDENCE is sourced, DESIGN is how the practice is designed, OPEN is not yet tested, and LENS is my own conviction.
You hand work to AI agents. Six human principles, observed in pilots, plant operators and managers decades before AI, predict where that handover breaks. They also predict something people get wrong: as agents get better, the human part of the job gets more important, and harder to keep sharp.
What this series covers: the human dynamics of agentic AI, in three layers
Most writing about AI agents is about the agents: what they can do this month. This series is about the people around them, in three layers that move at different speeds. Principles barely move; they held before computers and will hold after agents. Best practices move slowly, a few times a decade. Emergent trends move every quarter, and their numbers go stale fast. Part 4 applies all three to one question: how executives, startups and engineers can all win from the same work.
It's for leaders deciding how far to trust agents, and for the engineers who'll live with that decision.
Make a bet first
An airliner's autopilot gets twice as reliable. Does the pilot's job get easier or harder?
- A. Easier: there's less to do.
- B. No change: the pilot still flies the hard parts.
- C. Harder: the pilot does less, so is less ready for the moment the autopilot hands back control.
Hold your answer.
Principle 1: the more you automate, the more the human matters
EVIDENCE In 1983 the engineering psychologist Lisanne Bainbridge named the ironies of automation. Her central one: "the more advanced a control system is, so the more crucial may be the contribution of the human operator." And her sharpest: "a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one" (Bainbridge, Automatica 1983).
LENS Bainbridge wrote about process control, not cockpits, but her argument predicts C. Automation removes the routine practice that kept the human ready for the rare hard case, then hands the human exactly that case. LENS Every agent that "just handles it" is building an operator who will one day not know how.
Principle 2: reliance has to be calibrated, not maximised
EVIDENCE Raja Parasuraman and Victor Riley sorted the ways people get automation wrong into four: use, misuse, disuse and abuse. "Misuse refers to over reliance on automation, which can result in failures of monitoring or decision biases." Abuse is the designer's and manager's sin: automating "without due regard for the consequences for human performance" (Parasuraman & Riley, Human Factors 1997).
The goal was never maximum trust. It is appropriate trust: relying on the agent where it's reliable and checking it where it isn't. John Lee and Katrina See made that the field's definition two decades ago, and nothing about agents changes it.
Principle 3: delegation is a contract, and contracts leak
EVIDENCE Michael Jensen and William Meckling defined an agency relationship as "a contract under which one or more persons (the principal(s)) engage another person (the agent) to perform some service on their behalf which involves delegating some decision making authority to the agent." Their warning comes with a condition: "If both parties to the relationship are utility maximizers there is good reason to believe that the agent will not always act in the best interests of the principal" (Jensen & Meckling, Journal of Financial Economics 1976).
They were writing about managers and shareholders. LENS An AI agent breaks their premise: it maximises no utility of its own. But the gap comes back another way. The agent optimises exactly what you specified, which is rarely exactly what you meant, and you pay for that gap in monitoring, the same as any principal always has. An agent never pads its expenses; it just books you on the cheapest flight you technically allowed.
Principle 4: systems copy the organisation that builds them
EVIDENCE In 1968 Melvin Conway observed that "organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations" (Conway, Datamation 1968).
DESIGN A multi-agent system is a system in exactly that broad sense. If three teams each own an agent and don't talk, you'll get three agents that hand work to each other badly: the org chart, now with API keys. The fix is upstream of the code: decide who talks to whom, then design the agents.
Principle 5: a measure that becomes a target stops measuring
Charles Goodhart made the observation about monetary targets; the anthropologist Marilyn Strathern later gave it its familiar general form: once a measure becomes a target, it stops being a good measure. LENS Agents make this law bite harder, not softer. A person gaming a metric gets bored or guilty. An agent never does, and it searches harder than any person for the cheapest way to hit the number. Ask an agent to close support tickets faster and, unless you say otherwise, closing them is the fastest way.
Principle 6: people need competence, autonomy, relatedness, and safety to speak up
EVIDENCE Self-determination theory names "three innate psychological needs--competence, autonomy, and relatedness--which when satisfied yield enhanced self-motivation" (Ryan & Deci, American Psychologist 2000). Amy Edmondson added the team-level condition: "Team psychological safety is defined as a shared belief that the team is safe for interpersonal risk taking" (Edmondson, Administrative Science Quarterly 1999).
Handing work to agents touches all four at once. Competence: what am I still good at? Autonomy: who decides now? Relatedness: who do I work with? Safety: can I say the agent was wrong when my manager chose it? LENS An agent rollout that ignores these is likely to get compliance on the surface and quiet workarounds underneath.
One frame that ties them together: judgment, not just control
LENS "Human in the loop" gets used everywhere. EVIDENCE US weapons policy defines the idea most precisely. Its standard isn't control for its own sake; it is "appropriate levels of human judgment over the use of force," with "human on the loop" systems those "in which operators have the ability to monitor and halt" (DoD Directive 3000.09, revised 2023-01-25, as summarised by the Congressional Research Service, updated 2025-01-02).
LENS For agents, the useful question isn't "is a human in the loop?" A human can sit in the loop and approve everything without reading it, which is Principle 2's misuse with a signature on it. The question is where judgment is needed, and is the person there able to exercise it? That brings us back to the autopilot: able only if they've stayed in practice. Nobody wants the controls handed back for the first time in a storm.
The six, on one page
| Principle | Who named it | What it predicts for agents (my reading, not evidence) | |---|---|---| | The more you automate, the more the human matters | Bainbridge, 1983 | the person who checks the agent loses practice | | Calibrate reliance | Parasuraman & Riley, 1997; Lee & See, 2004 | over-trust and under-trust both cost you | | Delegation is a contract | Jensen & Meckling, 1976 | you pay to monitor the gap between spec and intent | | Systems copy the organisation | Conway, 1968 | agents hand work to each other as badly as their teams do | | Measures become targets | Goodhart; Strathern, 1997 | agents game metrics harder than people | | Competence, autonomy, relatedness, safety | Ryan & Deci, 2000; Edmondson, 1999 | rollouts that ignore them get quiet workarounds |
What we don't know
OPEN Some old laws may not survive agents intact. Fred Brooks found that adding people to a late software project makes it later, because of the time spent coordinating. Whether adding agents behaves the same way is an open question: agents add coordination cost, but not the months of onboarding. We don't know yet, and anyone who says they do is guessing.
How this could fail
- Principles as decoration. Six principles on a slide change nothing. Each one needs a check in the rollout plan.
- "Human in the loop" as a rubber stamp. If approvals are fast and never refused, the human is part of the automation, not a check on it.
- Treating the principles as anti-agent. None of them says don't automate. They say automate knowing what it does to the people.
Your move this week
Pick one task your team has handed to an AI agent. Ask three questions: when did a person last do it by hand (Principle 1)? Who checks the output, and how often do they reject it (Principle 2)? What number is the agent optimising, and what would game it (Principle 5)? Two answers mean trouble: "nobody has done it by hand this quarter," and a rejection rate of zero, which means the check isn't checking. If any answer is "I don't know," that's your first fix.
Next
Principles barely move. Part 2 is about the layer that does, slowly: the practices organisations have settled on for working with agents, and why two competing AI labs gave almost the same advice.
Sources
- Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6). https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf
- Parasuraman, R. & Riley, V. (1997). Humans and automation: use, misuse, disuse, abuse. Human Factors, 39(2). https://doi.org/10.1518/001872097778543886
- Lee, J. D. & See, K. A. (2004). Trust in automation: designing for appropriate reliance. Human Factors, 46(1). (Described, not quoted.)
- Jensen, M. C. & Meckling, W. H. (1976). Theory of the firm. Journal of Financial Economics, 3(4). https://josephmahoney.web.illinois.edu/BA549_Fall%202010/Session%205/Jensen_Meckling%20%281976%29.pdf
- Conway, M. E. (1968). How do committees invent? Datamation. https://www.melconway.com/Home/Committees_Paper.html
- Strathern, M. (1997). 'Improving ratings': audit in the British University system. European Review. (Described, not quoted.)
- Ryan, R. M. & Deci, E. L. (2000). Self-determination theory. American Psychologist, 55(1). https://selfdeterminationtheory.org/SDT/documents/2000_RyanDeci_SDT.pdf
- Edmondson, A. (1999). Psychological safety and learning behavior in work teams. ASQ, 44(2). https://web.mit.edu/curhan/www/docs/Articles/15341_Readings/Group_Performance/Edmondson%20Psychological%20safety.pdf
- Congressional Research Service (updated 2025-01-02). Defense primer: U.S. policy on lethal autonomous weapon systems (DoDD 3000.09, revised 2023-01-25). https://www.congress.gov/crs_external_products/IF/PDF/IF11150/IF11150.14.pdf
- Brooks, F. P. (1975). The Mythical Man-Month. (Described, not quoted.)