Recommendation · MoneyKeep receipts from every run — this value cannot be overemphasized. For an e-commerce app, save receipts of the entire user journey: login, page hits, errors seen, catalog load time, cart actions, and cart abandonment. Every run receipt can be fed to LLMs, or you can set up a system running all night long that aggregates the latest receipts. An overnight aggregation system can identify patterns like eight failures in the caching module and issue a PR that night to fix it properly.
endorsed
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyIf you have a web app and don't already have a custom verifier CLI, building one is one of the highest leverage things you can start with for your codebase. Also build a foundational eval that shows what good looks like in your application.
endorsed
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Anecdote · MoneyThe speakers describe that a fixed app can still lie: after reloading, the page could say 'waitlisted' while the screenshot says 'you're in, your seat is confirmed' — a mismatch that is a failure. Confirmation text that breaks the contract (not exactly one-to-one with what the model saw) causes the judge to fail even when the feature worked. They recommend avoiding tightly coupling verification to the exact text on the screen and proving things worked without that tight coupling, noting the answer is system-one models.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyIf your web app lacks a bespoke CLI that lets any agent fully flex the functionality you care about verifying, you'll have a much rougher time using prompts alone. Build a bespoke CLI (like their 'Snack' tool, run with pnpm snack) that lets any agent verify the functionality you care about.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyBuild a script that proves the work deterministically across a number of checks and feeds output back to the agent. Pin verification to a single command backed by actual bash commands because LLMs are non-deterministic and 30 agents with the same prompt produce variance. The agent can run the script repeatedly — run, check, verify, run again — until there are no more errors.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyRequire actual screenshots, videos, test results, and smoke tests as proof that an agent did the work. Accepting an agent's verbal claim that 'it passes, it looks good' without backing evidence leaves you with nothing to verify the work.
endorsed
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyGive your agents the ability to see all aspects of performance — networking, traffic, rendering, etc. If you're shipping a web app, give your agent Chrome DevTools. If you're shipping native clients (iOS or OSX apps), attach alternative observability projects, with more coming out every day. The common foundation across all platforms is that you need to give your agent the ability to see absolutely everything you would normally see as a developer. The more eyes you give your agent, the more it will be able to take over from you.
endorsed
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyDenys recommends having technical implementations and detection systems to safeguard your own data cleanliness. Unless you have good systems to detect duplicate records, you just keep getting the same bad quality data. Once you can detect duplications, you no longer incentivize things like sending claims multiple times with different dates or the same claim. As an organization, you have to have lines of defense in order to make sure that missing records get flagged.
AI Engineer · Insurance Claims Are AI's Messiest Data Problem — Denys Linkov, Wisedocs
Recommendation · MoneyWith claims data extracted, insurers can analyze what affects loss ratio, claim types, geographies, injury types, and doctors involved; map which lawyers, doctors, chiropractors, and physiotherapists work together; analyze which medical conditions co-occur (mapped to ICD codes) and compare write-ups from different doctors; analyze patient travel within provider networks; estimate treatment cost changes over time (e.g., a 10% year-over-year increase for a treatment type for a given doctor); and do counterfactual analysis on network changes, billing changes, and standard-of-care procedures.
Consequential creator advice requires independent verification before Today can promote it.
AI Engineer · Insurance Claims Are AI's Messiest Data Problem — Denys Linkov, Wisedocs
Recommendation · MoneyWhen a new model claim arrives via Slack or social media, Martone recommends not trusting public benchmarks. Instead, set the model up and run an internal experiment against your own benchmarks — ideally via a closed-loop system (e.g., a Kubernetes loop) that automatically benchmarks new claims on your own data.
endorsed advice
Also discussed by Stephanie Jarmak on AI Engineer, Zach Lloyd on AI Engineer (unclassified source — not counted until classified). Not counted as independent sources.
The a16z Show · Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone
Recommendation · MoneyThe agent's action log must be outside the agent's primary hands and brains because you can't trust the agent itself; you must be able to answer who did it, which is where the tamper-resistant auth log comes from.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyThe creator recommends putting the brain (model/LLM) in a sandbox with the tools inside it, and placing an egress proxy outside the sandbox as the authorization boundary. Building an agent loop with the brain inside the loop rather than outside the loop is easy to do and problematic. You can control what's inside the sandbox so it can't get API access on its own, and apply authorization at the egress proxy boundary rather than inside the agent.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyProduction agent teams should discover what's running and what the agent's capabilities are, validate inputs and outputs, and constrain agents using egress proxies and different networking rules.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyThe creator recommends keeping credentials and access separate from the agent itself so there is no broad standing access the agent has, noting that agents are deeply goal-seeking, so a random key found on a disk will be used by the agent.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyGive agents a token that is both time-limited and budget-limited; access ends when the token expires or the budget is exhausted, and a supervisor — not blunt rate limiting — decides whether to grant more budget.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyThe creator recommends giving agents a token that is time-limited and also carries a budget of how much resource they can consume, and providing continual feedback to the agent about how much budget was consumed and how much budget remains so it can operate within a budget like a person would. Traditional rate limiting just cuts an agent off and the agent doesn't know why it got cut off.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyIan recommends using broad-based deterministic policy that says what must never happen, or what always requires a human to be involved. He describes that there's a big gap between the policy you can write as a human or organization and what you actually want the policy to be in the specifics of a specific task, and humans can't write it all. He recommends layering deterministic policy with user-or-agent-to-agent interaction that carries transactional intent between the two, plus the business's intent for the specific task.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyThe creator recommends that there should be a separate supervising agent that decides what the worker agent gets to do, with separation of concerns: the supervisor can't do any of the work but decides what the worker does. Dick similarly recommends having a supervising agent that runs your policy and is independent of the agent doing the work. The creator gives examples: a request to read someone else's email gets caught in deterministic policies right away — it's not vague, and good controls exist for it. But telling an agent to forward the CEO's email to the outside world is vague, and ideally a supervising agent would catch it and say that's a bad idea and no, you can't do that.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyIan recommends that the auth log has to be outside the primary hands and brains of the agent because you can't trust the agent itself. He also recommends maintaining a record of who told the agent to do what thing, for what purpose, and whether that is still true — independent of enforcement points or vulnerability patching — and having the ability to answer the liability question of who did it, which is where the tamper-resistant auth log comes from.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyPut the agent's brain (the model/LLM) in a sandbox with the tools inside so it can't get API access directly, and place an egress proxy outside the sandbox as the authorization boundary — it does translation between systems, hosts OAuth, and applies authorization at the boundary.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth