Principle · MoneyDon't rely on the model itself to not do the thing you don't want it to do. Today's LLMs are far more probabilistic — that's their whole job — which is why you can't rely on model behavior alone.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Report · MoneyLotte Verheyden notes that teams not in high-risk industries can start with just tracing and monitoring, make simple fixes based on what they find, deploy to production, and iterate fast; the offline part (data sets, experiments, evaluation) makes sense once the product or setup matures, or in a high-risk environment where any mistake has bad consequences.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Principle · MoneyUse uniform conditions across every evaluation run so results are comparable. Changing any single variable in an evaluation setup makes it a new benchmark. Always do multiple runs when benchmarking — a single run is benchmark lottery: a random result and just a ballpark number.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Principle · MoneyIf the best and worst run scores are very close, there is not high variance and you can expect the same score. If they are far apart (e.g., 79% to 60%), there is a lot of randomness involved and you cannot really trust the model to execute the test even with a high score. A very high consistency score means the model is very reliable because it can be expected to solve the task all the time.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Principle · MoneyIt can happen that a cheaper and faster model takes longer and costs more than a bigger model because it just uses so many tokens. Before choosing a fast and cheap model, compare whether it is really faster and cheaper, because all those tokens also take time. Include cost as a metric when evaluating models.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Principle · MoneyWhen agents can generate thousands of lines of implementation detail, humans need a compressed review surface. A good diagram lets us see the shape of the system in a way code often obscures. Use diagrams not as decorative documentation after the fact, but as a way of making systems legible.
AI Engineer · Generation Is Cheap, Review Is Expensive: How to Stop Shipping AI Slop — Gabriel Martinez, G2i
Principle · MoneyUsers understanding privacy, their data, and how it's used is way more important than it always has been, as is users understanding the norms of your product. Before, you could get away with unpredictable behavior because there was a settings menu or a pop-up that could appear. When the agent is doing things semi-autonomously, you really need to make sure that at any given point a user can predict what is probably going to happen with a feature or primitive you're introducing.
endorsed
Lenny's Podcast · Stop planning for 2027: how OpenAI builds product 90 days at a time | Tara Sesha and Nan Yu (OpenAI)
Creator perspective · MoneyThe speaker argues that AI content all sounds the same and will feel like spam without human judgment, so the voice, taste, and judgment have to come from him. He also explains that automatically sending AI-generated emails leads to broken drafts reaching recipients — which is why the pipeline does not automatically send emails, and drafts are reviewed manually before sending.
Ahrefs · I Used AI Agents to Replace Our Marketing Agency
Principle · MoneyDesign AI agent products so the human reviews and approves the agent's plan before execution: the user states intent, the agent returns a plan, the user edits it, and only then does the agent execute. This keeps the vendor responsible for outcomes while making the right choices obvious to the user.
The Official SaaStr Podcast: SaaS | Founders | Investors · SaaStr 876: Shipping Enterprise AI Agents with the CPOs of Rubrik, Glean, and Harvey
PrincipleCharlie Guo's advice for GOAL: be really specific with the criteria you give the model for knowing when it's done, and make that completion criteria as verifiable as possible. When given a GOAL, the agent keeps evaluating itself against the success criteria, checking at each turn whether the task actually completed and whether to keep going. He compares GOAL to a genie: when it works well it is insanely magical, but when it doesn't it can be a monkey's paw situation where it technically did what you asked but not in the way you expected. GOAL suits very large projects — code migrations, large refactors, retry loops, experiments, games, and one-shotting apps with detailed specs — and he cites Thomas's slash GOAL that ran 40 hours reimplementing Doom in native Swift.
endorsed
AI Engineer · Cooking with Codex — Charlie Guo & Gabriel Chua, OpenAI
Principle · MoneyLaurie Voss calls reading your traces before you write evals the most important practice in the entire workshop: before writing a single evaluator, start with your data and read a dozen or more traces end-to-end. When reading traces, focus on the ones where something went wrong: what was the input, what was the output, what specifically is broken. Reading raw inputs and outputs sounds old-fashioned but is one of the highest value activities in agent development. A trace tells you whether the agent made a genuine mistake or whether your graders rejected a valid solution. Anthropic invested in tooling specifically for viewing eval transcripts, and their team spends a lot of time just looking at eval transcripts every day.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyThe speaker advised grading whether the output had the right information and whether the final state matched what you wanted, regardless of how the agent found the data; checking a specific sequence of tool calls is too rigid in practice, and grading the path punishes creativity because agents find valid approaches the eval designer didn't anticipate.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyWriting the eval first gives a clear, measurable definition of done; then build the feature until the eval passes. Some teams write evals at the very start, others add them at scale — either way treat evals as core.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Report · MoneyWhen an agent's system changes (model, prompt, tools) and behavior might drift, the solution is evals — specifically regression evals already in place before the change. Running them confirms the system doesn't change from previously good behavior, so you rely on evals rather than manual inspection.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI
Report · MoneyEval failure explanations can be fed back into a coding agent, which can then make the software better automatically and close the software development loop without manual intervention. Once you have defined what good looks like, your agent can climb the hill by itself.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI
Creator perspective · MoneyRob Walling argues that founders should validate an idea before committing to build, launch, and market it. Validation means doing research, putting thought in, having conversations, or testing the assumption that what you want to build is valuable. He frames this as a way to avoid spending months or years building something nobody wants, and notes that serial entrepreneurs do more validation before selecting an idea than first-time founders who flail. He also cautions that validation can never reach 100% certainty — you only get that by building something and selling it.
Startups For the Rest of Us · Episode 848 | What Liquid Death Can Teach Us About B2B SaaS (Rob Solo)
PrincipleReliability is the primary value driver for AI tools. Each additional 'nine' of reliability (e.g., 99% to 99.9%) unlocks new applications. Developers should prioritize background reliability over flashy demos.
The a16z Show · AI Can Write Code. Why Isn’t Software Better?
Creator perspective · MoneyA calm, unemotional energy that holds the line on agreements builds respect and makes deals work better, unlike a 'no problem' calm that lets things slide and leads to being walked over.
Charisma on Command · This Scene Will Change How You Handle Conflict
Recommendation · MoneyUse customer feedback as the most important way of judging how good an acquisition target's product is and whether its customers are likely to renew. Salesforce also used the target's own product as the diligence instrument — surveying Listen Labs' customers using Listen Labs itself, which proved both the product and customer satisfaction.
The a16z Show · Inside Salesforce’s $100 Billion M&A Playbook | John Somorjai
Recommendation · MoneyWhen deciding build vs. buy, assess whether your product team truly has the people and internal expertise to build the roadmap — or could easily hire it — before committing to build in-house. Compare a typical acquisition timeline of six to nine months end-to-end (idea to full integration) against how fast you could ship the capability yourself; an acquisition can be worth it when needed talent is very hard to hire, as with AI engineers today.
The a16z Show · Inside Salesforce’s $100 Billion M&A Playbook | John Somorjai