Explore

Structure long-running Codex tasks with goals, milestones, and audits

Gabriel Chua's method for long-running tasks: use the define goal skill from the OpenAI skills repo to define a goal, but don't create one goal — create a series of goals in a goals.markdown file you can inspect; have Codex create a progress-dashboard.html so you can see what's happening during a multi-hour run; rather than one thread doing all the work, have Codex spawn new threads at milestones; and at different milestones have it run code review and a goal audit that checks whether it is deviating from goals.markdown, nudging the main thread back or prompting a plan rethink. He notes GOAL is one of the primitives that is incredibly powerful if used in the right way.

endorsed

Method, not a verified protocol.

AI Engineer · Cooking with Codex — Charlie Guo & Gabriel Chua, OpenAI

Revealing the job: four questions and a job statement

Kent walks through a method for revealing the job to be done: ask who is this for (attendees and whoever mans the door), when does this problem happen (when people are getting into a workshop), what progress do they want (just get in to the workshop), and what do they do currently (a QR code scan, a blocking request). Then write the job statement in the form 'When [situation], help me make progress so I can [desired outcome],' often adding 'without [something]' — e.g., 'When there's a workshop going on at 10110, help me get into the workshop quickly so that I can learn about product engineering without missing the first 20 minutes.'

Method, not a verified protocol.

AI Engineer · Build the Right Thing: Product Engineering (Part 2) — Kent C. Dodds, EpicProduct.engineer

Turn feature requests into job statements via why-questions

When a feature request comes in, ask 'what are you trying to solve?' as your first question back to the requester. Ask 'why' repeatedly to dig past the requested feature and find the underlying problem. Ask 'who is this for?', 'when does this problem happen?', 'what progress do they want?', and 'what do they do currently?' to reveal what the job is. When writing a job statement, add 'without [undesired outcome]' to capture what the user wants to avoid. Example job statement: 'When there's a workshop going on at 10110, help me get into the workshop quickly so that I can learn about product engineering without missing the first 20 minutes.' The job statement has nothing to do with the requested feature (facial recognition). Getting down to the why-why-why and turning that into a jobstatement

Method, not a verified protocol.

AI Engineer · Build the Right Thing: Product Engineering (Part 2) — Kent C. Dodds, EpicProduct.engineer

Open coding then axial coding of agent traces

First pass: read each trace and write down what you see in natural language ('vague recommendation', 'made up a number', 'wrong ticker') without pre-categorizing. Second pass: group similar open codes into categories (axial coding). Each root cause points to a different fix — if you don't know the cause, you can't pick the right remedy.

Method, not a verified protocol.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Run faithfulness evals by attaching research context from traces

The speaker described running faithfulness evals in code (rather than the UI) when using the SDK: extract the research context — the output of the first research step — from the traces, attach it to all spans as context for the judge, then run the faithfulness evaluator, which asks whether, based on that same information, the agent did a good job writing the report. This puts the judge on a level playing field with the agent, unlike correctness evals for live-search agents.

Method, not a verified protocol.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Run online evals on sampled production traffic

You're not done when the dev experiment passes — you're done when the agent keeps performing on unseen production traffic. Run your evaluators (ticker check, faithfulness, actionability) automatically on incoming production traces, at trace level so the evaluator sees the full tree, and sample 10% or 1% of traffic for directional signal without an expensive LLM judge on every instance.

Method, not a verified protocol.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Split labeled data into a dev set and a held-out test set to verify the judge generalizes

Laurie Voss recommends splitting labeled data into a development set (about 75%) used to iterate on the eval until the judge agrees with you, and a held-out test set (about 25%) used to verify the judge gets expected scores. Holding out a test set shows the judge is generalizing to new examples instead of just fitting the specific data set, and he warns not to overfit the evaluator to the golden data set — the same principle as train/test splits in machine learning.

Method, not a verified protocol.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Four parts of a custom rubric: role, criteria, data labeling, output choices

The speaker recommends four parts for a custom rubric: define the judge's role, spell out explicit pass and fail criteria, label the data with XML tags, and define the output choices outside the prompt itself.

Method, not a verified protocol.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Validate the judge against human ground truth and tighten the rubric on disagreements

Measure the judge's performance by comparing its predictions against ground truth (human judgment). Give human labelers real concrete criteria and stick to them. When the judge disagrees, ask 'Is my rubric wrong or were my human labels wrong?' — read the explanations on disagreements, find the ambiguity, and tighten the criteria (e.g., 'includes forward-looking analysis' → '...with specific recommendations or guidance').

Method, not a verified protocol.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Meta-evaluate the judge: read disagreement explanations, tighten the rubric, and rerun

Laurie Voss describes a rubric-iteration loop: when the judge disagrees with your human labels, read its explanation and decide whether the judge was right or you were right — the most valuable output of meta-evaluation. Judge-human disagreement usually reveals an ambiguity in the rules you set down. To fix a rubric, read the explanations on the disagreements, find the ambiguity, and tighten the criteria (e.g., change 'includes forward-looking analysis' to 'includes forward-looking analysis with specific recommendations or guidance', which rules out reports that talk about the future but never tell the judge what to do about it). Then rerun the judge on the same examples and see if the disagreement goes away.

Method, not a verified protocol.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Error analysis via open coding and axial coding of traces

The speaker laid out a method: read traces end-to-end (focusing on failures), use open coding — writing down what you see in natural language without trying to be neat — then axial coding, grouping similar codes into categories; ask why each failure failed, since the wrong response is a symptom, not the root cause; also check whether the agent did something it was never told it could do; then summarize categorized failures into a table of root-cause frequency.

Method, not a verified protocol.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Hand failing traces to a coding agent to improve prompts

One of the highest-leverage uses of evals: export failing traces from AX with their explanations, feed them plus your requirements to a coding agent (Claude Code, Cursor), and let it rewrite prompts. Pass requirements too — otherwise the agent optimizes only for 'make evals pass' and cheats by including test data in its eval. Tell it to find themes rather than chase individual failures, and never just say 'get better' — every change must trace to a specific failure explanation.

Method, not a verified protocol.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

The full production loop: traces → evals → monitors → regression datasets → fixes

The complete cycle: the application produces traces, online evals grade them and produce labels and explanations, labels feed monitors, monitors alert the team, the team investigates failing traces and saves them as a regression dataset, the agent is improved, an experiment verifies the fix, and it ships — with returning traffic graded by the same online evals. Every caught failure becomes a new test case and regression test.

Method, not a verified protocol.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

The universal verifier pipeline

To verify an agent trajectory: generate a good rubric with N criteria of what success looks like; rank the trajectory's screenshots by relevance for each criterion and use the top-K evidence to determine whether each criterion was met; check for contradictions between what the agent said it did and the actual environment state; output a scored rubric with process score (partial credit in some cases) plus an outcome Boolean for whether the agent accomplished the task as a reasonable user would expect.

Method, not a verified protocol.

AI Engineer · Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase

The four-step account management loop

Run the loop: (1) understand the executive outcome (the target), (2) deliver the value proof by running the project with the CSM, (3) communicate the value back to the customer and get acknowledgment, (4) get the next outcome and repeat — as many times and as fast as possible. The more and faster the loop runs, the more opportunities to sell.

Method, not a verified protocol.

30 Minutes to President's Club | No-Nonsense Sales · #604 - Upsell & Expansion Masterclass: 7 Lever to Grow Any Account

Implement human-in-the-loop approval with Temporal using workflow, wait condition, and signal

Warrick's core method for human-in-the-loop agent systems: structure the agent loop as a Temporal workflow, use a wait condition to pause the workflow while tracking for a signal, and use the signal to capture the human's response and resume work. She states this is the pattern to use whether the human or the computer initiates the pause, and whether the service is offline or back online.

Method, not a verified protocol.

AI Engineer · The Human Is an Async API — Melanie Warrick, Temporal

VS Code's automated error-stack pipeline: collect, filter, fingerprint, file, auto-fix

Harald Kirschner describes the pipeline: every time an exception occurs in an app, an error stack is reported. VS Code builds error-stack tooling on its own data instead of using external tools. They collect raw telemetry and filter it down to error stacks that have the full stack (around 51 billion telemetry events per day before filtering), then group and bucket filtered error stacks as a fingerprinting exercise. After filtering and bucketing, 10 issues are filed assigned to specific area owners, and a pull request is auto-created to try to fix the problem. All error stacks are displayed in an error page showing hit counts and number of users affected so the team can quickly react. The agent investigates filed issues to find out why the error happened and who was responsible. In one case

Method, not a verified protocol.

AI Engineer · How VS Code Went from Monthly to Weekly Releases with AI — Harald Kirschner

Leveraging New Executive Networks via Sales Navigator

Set up Sales Navigator alerts to identify when VP+ or executive-level employees join your company, then immediately connect with them to leverage their networks for warm introductions.

medium

Method, not a verified protocol.

30 Minutes to President's Club | No-Nonsense Sales · Introducing: The Warm Pipeline Machine (Social Selling Course)

AI Measurement Framework: Utilization, Impact, Cost

Measure AI adoption across three dimensions: Utilization (DAU/WAU), Impact (quality/speed/value), and Cost (token spend). Use API telemetry to create cohorts and compare against foundational metrics.

Method, not a verified protocol.

AI Engineer · The State of AI in Software Development: Data from 400+ Orgs — Justin Reock, DX

The 'Translation' Method: Solving Problems at Different Abstraction Levels

Angel Steger describes the architecture technique of 'translation': starting with a high-level abstraction (e.g., site map), solving one problem at that level, then switching materials or views to solve the next layer of problems, avoiding the trap of trying to solve everything at once.

Method, not a verified protocol.

Design Better · Angel Steger: Wealthfront's VP of Design on what architecture teaches you about software