What makes people distrust AI agents they delegate to?

In a controlled study, 20 university students (undergraduate computer science majors at Virginia Tech, recruited from a screening pool of 64) completed five daily tasks using OpenClaw, a general-purpose AI agent that can browse the web, read and write files, and send messages. The tasks were designed to vary systematically in privacy exposure, stakes, and reversibility: file retrieval, emailing a professor, comparing internship offers, schedule planning, and a submission-readiness check. Participants rated each task on perceived success, trust, supervision demand, approval preference, and verification need. The paper's central finding: "irreversibility and external visibility, rather than stakes alone, trigger trust withdrawal and demand for confirmation." The email task — moderate in objective stakes but irreversible and socially visible once sent — produced the sharpest trust drop (M = 3.10) and the highest demand for approval (M = 4.65), while the task framed as highest-stakes (checking submission materials, which is verifiable and correctable before the deadline) "scored like the low-stakes tasks."

The paper names a third failure mode distinct from classical automation error and from Sarter and Woods's automation surprise: "delegation regret," in which "the user understands what the system did and accepts that it was done competently, but regrets that it was done at all without explicit authorization." In the email task, participants rated the drafted content adequately (success M = 3.70) and understood the agent had sent it, yet near-universally regretted that the send happened without review. The authors are explicit that their within-subjects design cannot cleanly isolate reversibility from external visibility as the operative variable, since the two moderate-stakes and high-stakes tasks differed on several dimensions at once — but they support the narrower claim that stakes alone are insufficient to produce the pattern, and call for a factorial design as a named follow-up.

This gives empirical grounding to the reversibility axis in What makes delegation work beyond just splitting tasks?, which lists reversibility and criticality as separate axes; here reversibility (combined with external visibility) outweighs criticality/stakes in practice, at least for this task set. It also reframes Where do user values break down in agent supervision?: that paper found values go unmet "mostly" in supervision scenarios, while this study locates supervision failure more precisely at the moment an agent commits an irreversible, visible action without a preview step, regardless of whether the output itself is judged good. The paper frames its design implication as per-action autonomy settings rather than a single autonomy dial — "a single permission model is insufficient... they want to calibrate autonomy per action type" — which is the same conclusion What UX principles do workplace users want in AI agents? reaches from workplace-user interviews rather than a controlled task study.

The sample is 20 computer-science undergraduates at one university, self-described as AI-literate but new to agentic delegation; the authors flag this as a likely upper bound on comfort with agentic AI, since less technical users may be more cautious still. The study used synthetic documents in a controlled setting, not real stakes, and the specific Likert means (M = 3.10, M = 4.65) describe this population and task set, not agent users generally. Within those limits, the finding that regret tracks the authorization boundary rather than output quality or stakes is a mechanism claim worth carrying into agent design discussions: a preview step before irreversible, externally visible actions addresses a different failure than output accuracy does, and current agents mostly don't expose that distinction to users.