Agent Build Log — Episode 006

Today, I started organizing the eval set for the Agent.

I’m going to build the first benchmark tasks from the skills I’ve accumulated over the years and the projects I’m currently working on. The initial focus will be common iOS and mobile development tasks, because those are the things I actually want the Agent to handle well.

Without evals, it’s hard to tell whether the Agent is genuinely improving or just happened to get one task right.

SWE-style benchmarks are not only about answering coding questions. They test whether the Agent can understand an existing codebase, identify what needs to change, implement the solution, run the tests, and actually finish the task.

I’m not building this eval set to chase a leaderboard. I want a baseline I can run repeatedly. Whenever I change the model, tools, or workflow, I need to know whether the Agent really improved—or quietly got worse.

That said, I haven’t created the first task yet.

For now, I’m having the Coding Agent review and categorize my existing skills and project work, so I can identify which real-world tasks are worth turning into repeatable benchmarks.

Before the Agent gets better, I need a way to prove it.