Creator perspective · MoneyThe speakers describe giving the agent a map with a verifiable destination — 'you must get here. You'll know you're here when this command returns this' — and note you don't necessarily have to prescribe how the agent gets there. If you prescribe an exact step-by-step path, agents may not sway from it, and you want them to sway a little while still reaching the right end goal; given a destination without a path, they may wander, catch more edge cases, figure something out, or ask a question that unlocks something.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyNick Nisi emphasizes that the value of keeping receipts from every verification run cannot be overemphasized. For an e-commerce app, receipts should cover the full user journey: login, page hits, errors seen, catalog load time, cart actions, and cart abandonment. Those receipts can be fed to LLMs, or a system can run all night aggregating the latest receipts and reporting failure patterns — for example, eight failures in the caching module — and a PR can be issued that night to fix it, with proof included in the PR that the issue is fixed.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyDavis says the more successful enterprises onboard agents like they onboard developers: agents do not get full commit access on day one and must add levels of capability as they mature. When determining an agent's autonomy level, enterprises should evaluate the context that makes the agent safe, including the environment around it, not just what the agent attests about itself. Whether an agent is running in a sandbox is a simple example of environmental context that can change the level of autonomy allowed.
None stated.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Creator perspective · MoneyDick believes there should be a separation of concerns between a worker agent that does the work and a separate supervising agent that decides what the worker agent gets to do; the supervisor cannot do any of the work itself. The supervising agent should be independent of the agent doing the work — it runs your policy and decides how things get done. Without an independent supervising agent, guardrail decisions about what an agent can and can't do get decided by the agent's vendor — and Dick does not want to cede that control. A request to read the CEO's email is outside the scope of an instance agent and should be rejected as intent from the user; a request to forward the CEO's email to the outside world is vaguer and should come back to a supervising agent, which would decide it is a bad
None stated.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Report · MoneyLaurie Voss argues you have to write down what good means before you measure it — you can't say your agent doesn't work if you haven't specified what working means. For a good financial analyst report: it should use real current data, be about the right company, stick to what it actually found, and (for his purposes) say what to do — buy, hold, or sell the stock — not just give analysis. Requirements sound obvious once written down, but before that they could be vague, or different people on your team could have different ideas about what a good agent would do. Writing requirements down turns vague disappointment into specific things that you can test.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss insists you have to read your traces before you write a single eval: read a dozen or more of them end to end — what was the input, what was the output, what specifically is wrong. Manually inspecting traces sounds manual and old-fashioned, but it's an incredibly high-value thing you can do when building an agent. You have to manually inspect what is going on so you can find the unknown unknowns that you weren't expecting. Anthropic's team regularly sits down and reads eval transcripts by hand and has built a bunch of internal tooling to make that easy to do.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Creator perspective · MoneyLaurie Voss describes the three eval types (code, LLM judge, human), their individual flaws, and recommends using all three simultaneously because they are complementary.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss explains that the span kind tells you what type of step it is — an agent tool, an agent, a tool call, an LLM, and so on. In tool calls and web searches, the span input is JSON, and that structured data makes the first eval easy to write. Spans also include timing and token counts, and often your agent will record what it costs — these are the fields to pull out and grade when writing evals.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyThe principles build on top of each other and form a logically sensible order: build a shared domain (plan mode) first, then set up consistency, then evidence, then accountability, then finally verification.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyTreating every input as a fact when some are estimates can invalidate a whole financial model. The model cannot tell them apart on its own, so you must enforce the distinction through harness design.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyInclude review gates on purpose when designing a harness so you can remain accountable for what is produced; without them, your only answer is 'the stochastic machine said it was fine.' Capture important judgment calls and disambiguations — not trivial ones — at those gates.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyThe main takeaway for designing accountability mechanisms is to use few gates with very high signal. Breaking tasks down in a way that you can reason about the issues and how they connect is a good example of a high-signal gate. You can be a little hand-wavy with reviews before functional tests pass, but you wouldn't push anything into production that hasn't passed all your tests.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyIf you don't make sure calculations run in code, the arithmetic is in the model's head and is gone once you start a new session. Make the harness strict so the agent doesn't do arithmetic by parsing tokens.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Creator perspective · BodyPlayers are bucketed into bullpen guys, position players, and starters, each with distinct training systems; bullpen guys are 'up' (back-to-back), 'down', or in between.
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
Principle · MoneySet up your test cases, evals, and examples, then do the search process from there. Find the cheapest model that passes most of your evals, then see how you iterate and fine-tune from there. It's also not that hard to just download 10 models from Hugging Face and try them as an alternative to using a provider.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Principle · MoneyYou generally only want to pay for tokens-per-second speed when the model is in front of someone (developers or customers). Background agents running at midnight don't deserve premium speed because no one is waiting for the output.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
ReportIf you ask an agent to write a poem or code and then ask it if the work is good, it will say yes because it did the work. Splitting generation and evaluation across separate agents (advisor pattern, generator-discriminator gap) gets better results; for massive problems, a generator-judge roster won't scale without separate and distinct checkpoints, and adversarial review should have its own isolated context uninfluenced by previous checks.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
Principle · MoneyThe speaker argues that removing one low performer (an 80) raises the team's average more than adding another high performer (a 140), so your bar will rise much faster by eliminating the duds than by adding new hires. He frames getting to a super high-performing team as coming from not letting people stay, and building a better team through elimination — get out everybody who sucks, and what you're left with is awesome. To increase the IQ per square foot, remove everything that is not high IQ. He notes this lesson requires boldness to act on.
Alex Hormozi · My honest advice on how to raise your team’s bar
Principle · MoneyWhen making hard people decisions, the speaker advises asking whether you want local empathy (for the individual) or global empathy (for the business and team). Shifting from local to global means you deal with the local pain while everyone else gets the global benefit; the reverse is that you get the local benefit and everyone else suffers the global pain. He encourages being willing to take the emotional pain of doing what's required to protect the high-performance culture you want to build.
Alex Hormozi · My honest advice on how to raise your team’s bar
Principle · MoneyThe speaker argues for a slightly easier way in, because judging people from one or two conversations and a document is limited, whereas judging their output is much better. Assuming someone has gone through training, you can just look at their output to evaluate them. He advises shying away from allowing someone to explain away a lack of output, noting that mediocre people are often good at selling their excuses.
Alex Hormozi · My honest advice on how to raise your team’s bar