Product pattern

Human in the loop

Keep a person able to review, correct or hand off work at the points where automation or a single user can get stuck.

Human in the loop Two versions of an AI assistant emailing 214 customers. In the first it reports that the email is already sent, with nothing to review. In the second it shows the draft and waits, with Approve and send, Edit and Cancel buttons. Acts first, tells you after AI assistant Done. I emailed 214 customers about the refund. “Your refund is on its way” Already sent to everyone Nothing to review, no way to fix it Shows the plan, you decide AI assistant Ready to email 214 customers about the refund. Send it? “Your refund is on its way” Approve and send Edit Cancel Able to adjust it, willing to use it Dietvorst, Simmons and Massey, 2018
An assistant that emails 214 customers unasked, beside one that shows the draft first and waits for approve, edit or cancel.

Human in the loop means the product puts a person at the right checkpoint: approving what an AI agent will do, adjusting its output, or routing a task to the colleague who has the credentials. On the site it covers explicit hold steps in AI flows, "Invite a colleague" on setup forms the current user cannot finish, and attribution that tells the team where a sign-up came from.

Parasuraman, Sheridan and Wickens (2000) proposed that automation can take over four kinds of work (gathering information, analyzing it, choosing an action, carrying it out) at different levels, and that the right level depends on the cost of errors. Dietvorst, Simmons and Massey (2018) found people often refuse an algorithm once they see it make mistakes, but were much more willing to use it, and forecast better, when allowed to adjust its output even slightly. Amershi and colleagues at Microsoft (2019) distilled 18 guidelines for human-AI interaction and tested them with 49 designers against 20 AI products.

In onboarding, the person signing up is often not the one who can finish setup. A built-in handoff turns that dead end into an invite, and in AI features a review step makes users willing to trust the output.

How to use it

  • Before an AI or bulk action runs, show what will happen and let the user approve, edit or cancel.
  • Let users tweak AI output in place (edit, regenerate one part) rather than accept or reject it whole.
  • Where a step needs someone else (admin rights, API keys), offer "Invite a teammate to finish this" with the context attached.

Watch out

  • Asking for approval on every trivial step creates alert fatigue, and people start clicking through without reading.
  • A review step that cannot actually change the outcome is theater and erodes trust.

Related

User control and freedomTransparencyNetwork effects

Human in the loop in real products 4 working · 0 not

Peliqan: Invite a colleague Peliqan (working) Invite a colleague Whoever opens this form often doesn't hold the credentials; Invite lets them hand it to someone who does. Julius: Explicit hold step Julius (working) Explicit hold step Step 2 says "don't do anything yet", a human checkpoint written into the prompt. Databricks: Tagged demo traffic Databricks (working) Tagged demo traffic "itm_data=demo_center" in the URL lets Databricks attribute sign-ups to the demo center. Lightdash: Hand off to the expert Lightdash (working) Hand off to the expert "If this is not you, invite them to Lightdash!" turns a stuck user into an invite.