A venture capital firm has built an AI that recommends which companies deserve a partner meeting, and its experience shows how AI evals cap what a product can do. NextView lets the machine read a company and make a call. Each week its partners grade what the machine said. Across a dozen revisions, agreement between the machine’s meeting calls and the partners’ own climbed from 50% to 92%.
Why the firm grades meetings, not returns
What the firm ultimately wants is investments that pay off for the funds’ backers. Venture returns follow a power law, most bets fail, and the verdict on any call takes a decade or longer. So the team aimed the machine at something gradable weekly: is this company worth half an hour of a partner’s time?
From weighted rubric to written instructions
Version one gave each company a weighted score across many attributes and added them up. Its first week of grading went badly: it pushed meetings the partners would have declined and passed on obvious ones. The score was complicated and hard to diagnose.
The team swapped it for plain-language instructions that describe what matters, leave the reasoning to the model, and ask for a verdict in the partners’ own format. Partners can now read why a call was made and point to where the logic went wrong.
How AI evals work
Picture training a junior investor. The junior does the analysis, a senior judges the same company alone, and then the two are compared. AI teams call this an eval. Each eval needs a grader, meaning whatever the output gets compared against before adjustments. That grader also caps how good the product can become.
There are three kinds of grader:
- Outcomes, when results come fast. A support agent can be judged by whether the customer’s issue got fixed. Coding and math improved earliest since software can verify answers and plenty of public examples exist.
- Expert answers, when outcomes are slow or absent. Legal AI company Harvey checks its agents against tens of thousands of criteria drafted by domain experts. Quality tops out at the level of whoever wrote the key, blind spots included.
- Expert reasoning, which is what NextView chose.
The problem with answer keys
An answer key only checks what its author thought to write down. The author’s example: ask for sausage, get pork when you wanted beef, and the output still passes because it is sausage. Add beef to the key and the model finds the next gap. It reaches the answer for the wrong reasons, and on a hundred other companies the right answers stop coming.
Grading reasoning avoids that, but only if the machine writes out its reasoning the way you reason. A weighted score isn’t how the partners think about a company, and reasoning you can’t read can’t be graded. With reasoning graded, a partner can follow a chain of logic to a conclusion nobody on the team would have reached and still see that it’s sound.
When to stop at the answer key
The author says most products should stop at the answer key. If you are replacing the routine junior slice of a job, a senior person’s answer is the right bar. Standard procedures have steps and a checkable result, so tune to the outcome.
Judgment work has no fast check. Researchers keep catching models that land on correct answers for flawed reasons and then break on new cases. Where a machine can verify the answer, the best results now come from grading outcomes alone. NextView doesn’t have that option, so it reviews reasoning, a bet the author says hasn’t been proven.
Why the human slice keeps shrinking
The final step stays human for now: the live meeting, where someone reads a founder’s visual cues, asks the follow-up and makes the final decision. That slice is expected to shrink. A machine has advantages a human junior lacks. The underlying model improves every few months. Every call is recorded with its reasons, so each new version can be checked against the full record. And as the companies it screened succeed or fail, those results become another grader.
What to watch next
In the near term a reasoning-graded machine and an answer-graded one will look alike. The test the firm describes is a machine suggesting a meeting no partner would have taken and then persuading them with its reasoning. Over a longer period the author thinks a machine taught to reason like a senior investor could beat the partners, though that is a guess, since human expertise keeps moving up as the model takes over lower-level work.
Source: NextView Ventures