Method · MoneyOnce you know who your audience is, figure out what they like. Associate your brand with what your audience likes by making content about the thing they like, making products they like (free and paid), and appearing next to the people they like.
Method, not a verified protocol.
Alex Hormozi · How I Would Build a Personal Brand in 2027 (If I Had to Start Over)
Method · MoneyStart AI evaluation with a clear requirement, such as 'advice must match the approved policy'. Challenge the AI chatbot with edge-case questions (e.g., about reimbursement after travel) to expose incorrect advice before deployment. If eval tests expose incorrect AI advice, add a policy check or human escalation as a control. After adding controls, test again to verify the controls catch errors before they reach the customer.
Method, not a verified protocol.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Method · MoneyStart by looking at your system claim and testing it to try to break it, then identify where the system claims break. Map out the uncertain region and high-risk region where the AI system's performance cannot be trusted, creating a full failure region map. Then either restrict the AI system from operating in failure territories or add supporting systems you are confident of, and iterate until reaching the desired confidence level. Apply this build-break-fix-break-again approach before deployment to expose failures that narrow evaluations miss in the real world.
Method, not a verified protocol.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Method · MoneyA first-principles approach for any evaluation system: start from your system claim and test it, trying to break it; identify where the system claims are breaking; when failures are found, either don't let the AI system run in the territory where there are failures, or come up with supporting systems you are confident of; keep iterating the build-break-fix loop until you reach the level of confidence you need. In practice: define the claim, actively challenge it, and use the evidence to guide changes and make a deployment decision.
endorsed method from the speakers
Method, not a verified protocol.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Method · MoneyUse a structured framework with 13 dimensions and three core questions — technical strength, deployment controls, and business context — to investigate AI failures. Technical strength covers data quality, task design fit, predictive validity, fairness checks, failure modes, and explainability. Deployment covers decision authority, human mediation, exposure control, and monitoring. Business impact covers financial impact, legal consequences, and scale of users affected.
Method, not a verified protocol.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Method · MoneyBreak your entire AI system into core components: technical failure, deployment controls, and business context. Assess your AI use case's materiality using a scale, which could be company-specific, and recognize that risk can change over time. Once you have the risk assessment, apply system-level stress testing on the AI use case. Only make the deployment decision once you are confident about the region; otherwise it may be too expensive.
Method, not a verified protocol.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
MethodTom Foley outlines an ordered method: first trace a typeface's calligraphic root (broad nib, pointed nib, or expansion), then compare structural proportions like x-height, ascenders, and cap height, then match stem weights and weight range/color/texture to narrow down pairings quickly.
Method, not a verified protocol.
Design Better · Tom Foley: How to pair typefaces and why the best brand fonts go unnoticed
Method · BodyThe coach uses a four-day training split where every day develops a different piece of the explosive power puzzle: day one focuses on increasing max lower body strength, day two on speed strength (possibly with upper body work), day three is 'athlete day' with lots of jumps and plyometric work, and day four is the impulse session focused on putting out a large amount of force in a short period of time.
Method, not a verified protocol.
Garage Strength · I Trained 1000 Elite Athletes. This Is How You Jump Higher.
Method · MoneyInstead of only grading agent answers, plant known problems or answers into inputs and test whether the agent finds them back. Mix modified contracts (with removed clauses) with existing unmodified contracts that should show no flags, so the agent must distinguish real defects from normal contracts — the team found their agents were missing most removed clauses.
Method, not a verified protocol.
AI Engineer · How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Method · MoneyWhen pairs of original and modified contracts cannot be obtained, pivot to generating test cases from known rules: take a rule (e.g., a $1 million cap), create a clause that breaks it, generate the contract around it, and verify the agent flags the violation. Source rules from experts and their documentation, supplement with your own rules, and design generated tuples so some break rules and some do not — full rule coverage cannot be guaranteed.
requires knowing the rules; coverage of all rules cannot be guaranteed
Method, not a verified protocol.
AI Engineer · How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Method · MoneyReuse the templates experts already use, which contain blanks; generate an array of values for each blank to produce plenty of test contracts. The speaker identifies this template approach as the most powerful technique, while noting it is very hard.
speaker notes the technique is very hard
Method, not a verified protocol.
AI Engineer · How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Method · MoneyRun a weekly skill that reviews your video creation process, shows where you spend the most time, and suggests repetitive tasks to automate. Use it to identify which skills need updating because they're not being effective, consolidate your task triage into one place (it identified reminders being sent to a Slack thread that was never checked), and remove unnecessary recurring checks (it identified a Premiere import check that took a lot of time and wasn't needed).
Method, not a verified protocol.
AI Engineer · The AI Content System Behind 200M Views — Sarah Chieng, Cerebras & Cova Lee, OpenAI
MethodRate seven highly polarizing films on your own personal taste, then find which critic's ratings best align with yours to identify a trusted critic for future recommendations. Use the nine films that top critics most disagree on as the calibration set, and watch the polarizing calibration films you haven't seen before rating them. When building the critic shortlist, exclude critics who don't publish numeric ratings (prose-only or video-review-only critics), since the comparison needs numbers. Accept that even a taste-matched critic will sometimes disagree with your ratings — that's expected and fine.
Method, not a verified protocol.
Slidebean · yeah... Netflix only wants you to watch THEIR movies
Method · MoneyLevel one of the review method prevents agents from overstating what they did. Before an agent starts a job, the job spells out exactly what it has to hand back: the file, where it goes, and a command that checks it. The job also points out shortcuts agents have taken before so the current agent doesn't repeat them. The agent sent the job runs the check itself, and the reviewing agent must end its review with either 'artifact verified' with a fingerprint of the file, or 'artifact missing' with the reason. Saying 'done' on its own is not a permissible answer.
Method, not a verified protocol.
AI Engineer · I Believe Everything My Agents Tell Me: Three Levels of Review — Corey Gallon, Rexmore
Method · MoneyLevel three of the review method prevents agents from getting the reasoning wrong. When clarity of thought really matters, perform adversarial review in three stages. Stage one: the agent audits its own work, listing every claim and marking each as measured, checked against the source, inferred, or assumed; anything it can't back up gets tested or labelled as a guess. Stage two: reviewers with none of the original agent's context, each acting as a persona with reasons to find fault, read the work cold — each plays a real reader and gets the work and the sources, but no history, drafts, or conclusions to agree with; their findings get checked against the source before anything changes. Stage three: once the work is revised, brand new reviewers who've never seen it check the final version.
Method, not a verified protocol.
AI Engineer · I Believe Everything My Agents Tell Me: Three Levels of Review — Corey Gallon, Rexmore
Method · MoneyLevel two of the review method prevents agents from getting the facts wrong. When correctness matters, first find something in the source that the answer has to agree with. Two agents get an identical set of instructions and do the same job independently, neither seeing the other's work; each notes where every fact came from and checks its results against the facts. A third agent with the original source in front of it compares the two agents' work line by line and settles every difference by going back to the source, never forcing a match.
Method, not a verified protocol.
AI Engineer · I Believe Everything My Agents Tell Me: Three Levels of Review — Corey Gallon, Rexmore
Method · MoneyAl Chen describes a workflow for CVEs: create a skill that does the heavy lifting of CVE reporting and internal research, outputting a table of what to fix and explanations for bank security teams; create a Harbor task with instructions, Docker images, and tests; instruct the agent to write an exploit against the old version and a test to verify the new version remediates it; and run many Docker containers or Modal/Daytona sandboxes in parallel to get trials of agent performance. The point is that bumping a package version is not enough — you must provide evidence the exploit is gone.
Method, not a verified protocol.
AI Engineer · Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco
Method · MoneyCompanies that successfully ship and continuously improve reliable agents follow a set of practices called the AI engineering loop. The foundation of the AI engineering loop is always tracing. Monitor aggregate metrics on what is going on and detect signals where things are going wrong. What you get from production through monitoring is one of the foundations you can use to build out your data sets. Run experiments by trying different variations of your system to check how it performs against your data sets. Continuously close the AI engineering loop: learn from production and feed it back into data sets.
endorsed
Method, not a verified protocol.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Method · MoneyKormos lays out a phased rollout: on day one, start supervised — agents and humans work together as part of the process, and output is manually verified to ensure it is of high quality. During the supervised phase, refine data sets and ensure the right confidence mechanisms are in place, with domain subject matter experts defining what good looks like and beginning to verify it. When measured confidence reaches a policy threshold, move towards sampling: humans look at a slice of the output rather than all of it. During sampling, begin systematic evaluation through traces to ensure what the humans aren't seeing still meets the quality bar. Once the confidence threshold is cleared, humans handle more of the exceptions of the process but still act as participants in refining the data sets and
Method, not a verified protocol.
AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase
Method · MoneyFor traditional model backtesting, gather the data needed to backtest the model you already have, even if the live system doesn't use agentic web search. Use Snapshot to jump back to a known URL or website and extract whatever feature is needed (e.g., an array of numbers), even if Exa isn't involved in the prediction itself. Backtest agentic features one by one against true positive matches to see whether the system correctly predicted that a business succeeded or didn't succeed. Add agentic features into your system one feature at a time so you can control the experiment, and tune non-deterministic systems carefully because the way you tune them is really important — searching one way may succeed more often, another way less often.
Method, not a verified protocol.
AI Engineer · Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa