Report · MoneyTo keep evals cheap, start with actual deterministic tests (unit tests). If deterministic tests can't be done, the next level up is regular expressions or pattern matching. If pattern matching fails, the next thing to do is examine your traces. Ideally, everyone is using some type of observability solution and has traces of each message envelope, inputs, outputs, tool calls, etc., which can be validated per object. Eventually you will come to a place where you need to use a judge, and then you start with the cheapest model. Evals are afforded strategically, because it will not be cheap if you use the most expensive model, and you want to build up to it. Regular expressions can be used for testing AI outputs but have their limits.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyWhen humans and AI don't agree, it's usually because of a knowledge gap — and the way to reach agreement is to introduce context to bridge that gap. To improve judge agreement, examine the cases where you disagreed and fill the knowledge gap with policy. In production you would supply policy to the judge via a RAG pipeline that retrieves enterprise context; in a demo you can just hard-code the policy. Write the policy explicitly (e.g., a 14-day return window; defective items can be returned) and interpolate it into the judge prompt. Building your judge is arduous iterative work — investigating failures and filling knowledge gaps is literally the job. After adding policy, check whether agreement is growing even if it still fails — improvement indicates you're on the right track. When the
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Principle · MoneyThe minute the account is yours, engage right away so the value loop starts turning — the AM not being involved in implementation is a mistake. At handoff, verify (don't assume) the executive outcomes the AE used to win the deal, grab them, keep the CSM on target to them, and deliver those as the first set of value; that proof is what lets the next outcome happen.
30 Minutes to President's Club | No-Nonsense Sales · #604 - Upsell & Expansion Masterclass: 7 Lever to Grow Any Account
ReportThe environment.id parameter enables routing to the exact same context so you can use the same sandbox. As long as you preserve the environment ID, it will return you to the original sandbox you had. Any packages installed or files created remain accessible to the model through each turn, as long as it passes back the interaction ID and the environment ID. You don't need to manage any persistence of state yourself; you only need to add a few lines of code.
AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind
ReportWith the Interactions API you can mix and match different inbuilt tools, which wasn't possible before. Example agent setup: give the model Google Search (the same index Google has), the URL context tool to retrieve information from web pages, and a custom tool called File Incident, to look up the latest security reports for a React application. With mixed tools, the model can reason autonomously, figure out what information it needs, and generate a response — all in a single API call. Giving models internet access makes them powerful because they can work with information beyond their training cutoff date.
AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind
ReportWith the responses API, the first turn (e.g., 'hi, my name is Phil') returns an interaction ID; passing that ID back as previous_interaction_id on the second turn preserves all context. The interaction.id parameter lets you preserve and work with context and ensures you're not busting your cache.
AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind
ReportIn the past, building a coding agent required queueing in the harness, finding a sandbox provider, managing the infrastructure, and figuring out how to preserve context between runs. With the anti-gravity managed agent, a single API call gets you a persistent sandbox out of the box that you can keep hitting and treat as a personal cloud without any modifications.
AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind
PrincipleIndustry self-regulation (e.g., Hollywood's MPAA) can prevent heavy-handed government intervention. However, if risks are deemed 'non-zero' for catastrophic outcomes, the logical regulatory response is nationalization, as seen in critical infrastructure sectors like banking and power.
medium
The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?
PrincipleAI safety discussions should be grounded in concrete, identified risks (e.g., novel cybersecurity threats) rather than speculative super-intelligence scenarios. Security claims should be validated using physics-based and systems-based constraints (e.g., entropy, bit rates) rather than abstract 'super-intelligence' capabilities.
The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?
PrincipleThe center of AI innovation has shifted from model creation to the integration of models into traditional software systems. This involves using language models to return probabilities for conditional logic (e.g., 'if X percent') rather than binary true/false checks, reviving 1960s-70s probabilistic programming research.
medium
Commercial context: The creator presents this as their own offering and directs viewers to get it.
The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?
PrincipleAI agents should be treated as untrusted entities with oracle-like capabilities in threat models. Unlike human employees, they can execute thousands of security probes rapidly without fatigue, requiring granular access controls and monitoring of internal APIs for denial-of-service style attacks.
The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?
Method · MoneyKanazawa describes a loop: create rubrics with an agent, hill climb to the best possible outcome, then iterate on the rubric itself so the outcome gets better. He sets the rubric, lets the agents hill climb to optimize toward it, then reviews and iterates.
Method, not a verified protocol.
AI Engineer · I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. — Keiji Kanazawa, Microsoft
Method · MoneyBen Holmes describes a coherent loop: (1) grade conversations as soon as they happen, e.g. on a cron job every 24 hours going through a sampling of the past 24 hours of conversations; (2) aggregate pass and failure rates over time across agent conversations; (3) have an agent describe why each failure or pass happened, so the explanation becomes context for another agent; (4) filter graded conversations down to the failures and pass the failing reports — which describe what failed and why — to a self-improvement agent, telling it to find commonalities; (5) give the self-improvement agent access to the skills repository and the code base if you want it to make architecture changes; (6) continue evaluating after improvements to see how the scores perform. A self-improvement agent might, for
hedged
Method, not a verified protocol.
AI Engineer · Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp
Method · MoneyVercel's GTM engineer shadowed one of their best SDRs for a few days, watching every screen and tab (LinkedIn, company website, CRM, Slack history). The engineer converted each data source into a step in a tool-calling workflow, documented it, and made the workflow leverageable before touching any AI. After building the first version, the agent ran in production in shadow mode for six weeks while the best SDR reviewed every single output to create a feedback flywheel. The human QA reviewer was pulled from the loop only when the subject matter expert had less feedback for the agent over time.
Method, not a verified protocol.
The Official SaaStr Podcast: SaaS | Founders | Investors · SaaStr 885: Going from 10 to 1 SDRs. How We Deployed AI Across our Sales Org with Vercel's COO
Method · MindHuberman prescribes keeping total training volume low — about two exercises per body part, maybe three — so recovery is fast, the nervous system isn't overtaxed, and progress stays high. He instructs writing down what you did in every workout and increasing either weight or repetitions every single workout, for every body part.
hedged
Method, not a verified protocol.
The a16z Show · Andrew Huberman’s Three Rules for Better Health and Focus
Method · MindHuberman prescribes a structure for each resistance-training exercise: start with one light set of 10-12 repetitions just to get blood flowing, then a moderate set — a weight you could do 10 repetitions with but only do 3 — to prepare the nervous system for heavier load, then only two all-out work sets taken to momentary muscular failure in perfect form, spaced 3-4 minutes apart. He says people go badly wrong by over-warming up, and that a light set and a moderate set are sufficient preparation.
hedged
Method, not a verified protocol.
The a16z Show · Andrew Huberman’s Three Rules for Better Health and Focus
Method · MoneyThe process the Instagram team followed was to ask a very clear root cause of why you were not able to do the thing that was the goal — the inverse of asking why you did the thing. In the Instagram Stories root-cause exercise, people answered the why-not question, and when the answers were grouped by theme, the top theme emerged. When analyzing why users did a desired action, ask 'why did you do the thing' — the inverse of asking why they didn't. Group open-ended user answers by theme and sort them to find the top root cause. Follow qualitative root-cause interviews with a large quantitative survey of thousands of people to quantify the weights across themes.
Method, not a verified protocol.
Lenny's Podcast · What it takes to be a top PM today | Robby Stein (Google Search)
Method · MoneySagar Batchu describes deciding in real time whether a call succeeds or fails as three parts. R (credentials) is the ceiling of what a human put down when they issued the key — it never gets widened, lives on the credential, and can't be escalated past. A (the agent's own policy) can't hold admin privilege or any human-only scope. O (the owner's live policy) means an agent can never do more than the person accountable for who's actually driving the agent. A and O are read on every call, so removing a permission takes effect on the next request. Applied to the database scenario: an agent scoped to staging can't delete production even if the owner could, and an agent can't exceed an owner who never had production rights — either way the call stops. And if an owner leaves the company, the IDP
Method, not a verified protocol.
AI Engineer · A Valid Key Let an AI Agent Delete Production - Sagar Batchu, Speakeasy
Method · MoneySagar Batchu recommends modeling a governance architecture around four steps. Connect: create a door instead of dozens of integration points — all agent calls to external systems go to one place, like an MCP gateway. Resolve: calls are resolved to a named agent or human behind it, since policy evaluation doesn't work without identity, and the identity must be the same one all other enterprise software is assigned to. Decide: allow, deny, quarantine, or escalate — sometimes just warning a user is sufficient. Record: the decision is recorded before any action is actually taken.
Method, not a verified protocol.
AI Engineer · A Valid Key Let an AI Agent Delete Production - Sagar Batchu, Speakeasy
Method · MoneyKarri Saarinen describes a feature roast practice: attendees critique the whole feature with raw, non-personal feedback ('you can be as nitpicky or confused as you want'), framed as 'This is how I see it' rather than a personal attack. The building team can interact with attendees and ask questions when they don't understand the feedback. Afterwards, the lead synthesizes the feedback, groups it, and turns it into actionable issues to fix.
Method, not a verified protocol.
Lenny's Podcast · Context is now the product: Product leadership when software can build itself | Karri Saarinen