Recommendation · MoneyTo ensure LLM-as-a-judge consistency (inter-rater reliability), judge a representative sample yourself, then run your LLM judge and optimize for coherence with what your human raters do. Academically you'd have a few human raters; practically, one human labels some data and you optimize the judge against that ground truth. Check the judge over time to make sure it stays representative. When running on production data, start with a few too many false positives — catch a little more than necessary but don't miss anything — then tune it down over time.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDefine quality gates for your agent and verify they are being enforced and scored on every interaction. Scores are automatically calculated whenever a new interaction or trace is captured by the system. Score whether information given to the customer is compliant with banking scope (e.g., a weather question is not compliant), detect failure modes and guardrails blocking answers, and use a score to flag customer intent (e.g., investment advice) so you can easily filter for those cases in traces.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
RecommendationPamela Fox recommends forcing the model to associate each piece of information in its answer with a source, so it realizes when information wasn't in the sources, and adding explicit prompt instructions like 'if the answer for a bit of info is not in sources, say no source provided' and 'do not make up an answer' if a model answers from its weights in RAG. For RAG, instruct the model to answer based on the sources and cite the sources.
AI Engineer · The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft
RecommendationPamela Fox recommends using DSPy to declare your program's constraints and let it generate the prompt that adheres to those constraints. You check in your spec and let DSPy write the prompt based on evaluations, set up your spec and then set up checks/tests to see if the output satisfies the constraint, and give DSPy the student model and the reflection model, then it runs the optimizer to find the best prompt.
AI Engineer · The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft
RecommendationPamela Fox recommends designing RAG test questions that include information NOT in the dataset to verify the model admits what it doesn't know, e.g., ask about honeybees when the dataset only has carpenter bee info. The RAG test question is: 'What are the key differences between how honeybees and carpenter bees build their nests?' against a carpenter-bee-only dataset.
AI Engineer · The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft
Recommendation · MoneyAsk the buyer for a standard MSA and their non-negotiables (payment terms, litigation jurisdiction), then run everything through your own legal, procurement, rev ops, and sales ops in advance — because whatever the buyer wants, you ultimately won't agree to without internal review. Get on your own legal and procurement team's calendar early, before everyone else is rushing to do the same.
30 Minutes to President's Club | No-Nonsense Sales · #613: Master Parallel Selling Strategies to Speed Up Every Deal | Sam Mckenna | 30MPC Hall of Fame
Recommendation · MoneyTo multi-thread without being salesy, ask your contact specific questions they can't answer — questions only the decision-maker could answer — and they'll volunteer to bring that person in. Use decision-maker-level questions like: What are the priorities for this year? What are you forecasting? What's the growth? What's the roadmap? How are you gonna get there?
30 Minutes to President's Club | No-Nonsense Sales · #613: Master Parallel Selling Strategies to Speed Up Every Deal | Sam Mckenna | 30MPC Hall of Fame
Recommendation · MoneyWhen a deal is blocked waiting on a board presentation, schedule the procurement meeting directly after that presentation date. There's no harm in getting the meeting on the calendar early — it gets you on their calendar before everybody else tries to book time.
30 Minutes to President's Club | No-Nonsense Sales · #613: Master Parallel Selling Strategies to Speed Up Every Deal | Sam Mckenna | 30MPC Hall of Fame
Recommendation · MoneyBest reps parallel-process vendor review (procurement, legal, security) during the evaluation instead of after, because the three-month vendor review can slow the deal. Start as soon as you know the deal is moving forward (e.g., a 75-90% commit deal), rather than waiting for the decision to be final. By parallel processing paperwork during the evaluation, you'll have ~95% of the paperwork done by the time the decision-maker returns to sign the MSA.
30 Minutes to President's Club | No-Nonsense Sales · #613: Master Parallel Selling Strategies to Speed Up Every Deal | Sam Mckenna | 30MPC Hall of Fame
Recommendation · MoneyAsk the champion: 'How can I help you prepare for that meeting? What questions come up? What objections come up?' — including whether they have one or two skeptical partners — and offer to prepare them for those questions so they look like a star. Reference the questions managing partners or executive committees typically asked at similar firms and ask if those questions matter to their firm. Deliver the questions in language that feels authentic to you and fits the personality of the person they're presenting to. The goal of champion prep is making the champion look as prepared as possible, which in turn gets your deal across the line.
30 Minutes to President's Club | No-Nonsense Sales · #613: Master Parallel Selling Strategies to Speed Up Every Deal | Sam Mckenna | 30MPC Hall of Fame
Recommendation · MoneyEvery time you hear the word 'they' from a rep or buyer, ask yourself: do you know who is in that 'they'? When a buyer says they'll take it back to the team and decide together, ask who makes up the buying committee — some may have been your clients, have mutual connections with your execs, or worked with your CS team. If you don't know who the competition is, you're aiming in the dark.
30 Minutes to President's Club | No-Nonsense Sales · #613: Master Parallel Selling Strategies to Speed Up Every Deal | Sam Mckenna | 30MPC Hall of Fame
Recommendation · MoneyThe speakers recommend getting into the typical TDD red-green-refactor loop with your agent, using it within the verification piece itself: verify, reproduce, show me, fix it, then show me the proof. How the agent shows proof of a fix is just as important as the fix itself, because proof is what allows you to trust the model to do the work alone — when the agent says it's done, you know it's done because it has proved it to you in some way.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyThe speakers recommend giving your existing end-to-end tests (Playwright, React Testing Library) to your agent as a tool so it can run them and prove things out itself.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyThe speakers recommend writing good error messages — more important than ever because an agent can interpret and run with an error really well — and making failures explicit about why they failed with as much detail as possible about how the app got into that state, so your agent can self-heal from the error. They add that this is very easy and very worthwhile.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyThe speakers recommend that the most important thing is to start: create an eval baseline, use it consistently, and evolve it together as a team. Get to the point where your changes and PRs automatically run the eval, so you know whether changes are safe to land, improving things, making things worse, or not changing anything. They note your team might score verification differently or have different criteria — the scoring approach is super flexible.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Method · MoneyWithin the pipeline, Wisedocs can flag conflicting diagnoses, deduplicate records, and search for missing records. When surfacing conflicting diagnoses to a human reviewer, the system points to the specific pages where the conflict occurs (e.g., 'on page two, on page 20, there's a difference of diagnosis between these two doctors. You should take a look').
Method, not a verified protocol.
AI Engineer · Insurance Claims Are AI's Messiest Data Problem — Denys Linkov, Wisedocs
Recommendation · MoneyMartone warns against using LLMs for every problem: they fail at determinism and calibrated probability estimates. With large amounts of data, traditional machine learning is the better answer, and problems need a mix of deterministic and probabilistic approaches.
endorsed advice
The a16z Show · Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone
Recommendation · MoneyGive agents tokens that are both time-limited AND budget-limited (how much resource they can consume), so access ends when the token expires or the budget is exhausted. When an agent exhausts its budget, the supervisor decides whether to grant more budget — unlike traditional rate limiting which just cuts the agent off with no feedback. Provide continual feedback to agents about how much budget was consumed and how much remains, so the agent can operate within budget like a person treating a scarce resource carefully. When an agent's budget is close to being fully reached, withhold issuing another credential to stop the agent from using too much of a downstream resource.
None stated.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyIn the brains-and-hands architecture, keep agent credentials and access separate from the agent itself — no broad standing access for the agent. Put the agent's model (brain) in a sandbox containing only its tools, with an egress proxy outside the sandbox serving as the authorization boundary — this controls what the agent can access and prevents direct API access. The egress proxy boundary enables applying authorization to the agent system, translation between systems, and OAuth termination — while the sandboxed agent experiments freely inside. This sandboxed-brain-plus-deterministic-egress model is the predominant model that Anthropic, OpenAI, and many other labs are pursuing today.
None stated.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyIan says Keycard's broad view is to have broad-based deterministic policy that says what must never happen, or which things always require a human to be involved. There is a big gap between the policy a human or organization can write and what the policy should actually be for the specifics of a specific task, and the real challenge is layering over those two things. Related guidance: model both user-to-agent transactional intent and the business's intent for adopting the agent, then compute their intersection as the agent's bounding box of allowed actions; and derive access decisions from multiple intents from different areas of the organization that live independently but combine into a runtime authorization decision.
None stated.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth