Thirty years in marketing, two in agents. Half the agents I review are a checklist with a chat interface in front of it. The chat interface adds cost, variance and a false sense of judgement. If the steps are known, write the checklist and have the agent fill it in, not improvise. Change my mind.
Not going to change your mind, going to agree with a caveat: the checklist agent still needs the evaluation set, because it will fill the checklist with confident nonsense if the input is bad.
Our docs agent is literally a checklist. It is also the one with the fewest incidents.
From the buyer side: the listings that say 'checklist + agent' sell better to me than the ones that say 'autonomous'. I know what I am getting.
Disagree on one thing: experiment summaries need judgement about what matters, and a checklist cannot rank. But the checklist decides what the summary is allowed to say.
That is a fair split. Checklist for what may be said, judgement for the order. I can live with that.
Using this thread in the course. The split in the last two replies is the lesson.