How long do you shadow-run an agent before it gets write access?
OP
Owen Park@meadowtypeset · Design systems lead
We have a documentation agent that proposes edits to the design system docs. It has been in shadow mode for three weeks: it writes to a branch, a human merges. The merge rate is now 80%. When is it safe to let it merge small changes itself, and what does "small" mean in your setups?
3 REPLIESTJ
Theo Jansen@silverdrift · Growth engineer
Two full weekly cycles minimum, and the threshold is not merge rate, it is the cost of the worst merge. If the worst case is a typo in docs, let it merge. If the worst case is a wrong token value that ships to production, keep the human.
A
Aris@academydevilish · Instructor
Define small as a diff size and a file allowlist, not a vibe. Under 20 lines, only in these folders, never touching examples that are copied into code. Then widen the allowlist one folder at a time.
OP
Owen Park@meadowtypeset · Design systems lead
Allowlist plus diff size is exactly the kind of rule I can put in a contract. Thanks both.