MLCommons has launched MLPerf Endpoints v1.0, a benchmark that measures live AI API endpoints under production-like load instead of testing a tuned chip in isolation. MLPerf Endpoints evaluates the whole stack together: hardware, model, serving software, batching behavior and networking.
MLCommons is the open engineering consortium behind the MLPerf family of AI performance tests. Earlier MLPerf benchmarks focused on silicon. This one targets what buyers actually use.
Why a new benchmark
For cloud providers, hardware makers, managed services and neoclouds, existing benchmarks can turn into a tuning race that says little about business value. A score shows how well a system did on that specific test and nothing more. MLCommons compares it to buying a car from the top speed printed on the odometer without a test drive.
Endpoints instead produces a full Pareto frontier of system behavior across workloads, not a single peak figure. No data is estimated or interpolated. Only measured conditions appear, including the points where customers could hit a performance cliff.

The LoadStair visual
Results are shown in the MLPerf LoadStair Visual. It plots discrete, measured operating points as step functions, so readers see where an endpoint’s behavior changes under scaling demands. Traditional charts use smoothed curves that guess at a system’s limits.
MLCommons pitches several benefits to providers:
- A credibility stamp backed by peer review from industry competitors and third-party auditing
- Credit for full-stack work, such as a custom scheduler or an optimized networking fabric, which narrower benchmarks ignore
- Workloads customers run today, including multi-turn AI agents, instead of synthetic single-shot tasks

How to submit
MLCommons says the benchmark lowers the engineering burden of earlier rounds. Submissions roll continuously, with publication cohorts every few weeks. The process runs as follows:
- Sign the MLCommons Contributor License Agreement and get a PRISM API key through an MLCommons membership account.
- Install the reference clients.
- Plan the Pareto frontier by choosing concurrency boundaries. Agentic benchmarks need exactly 7 measurement points. Non-agentic suites need 8 to 32, including a dedicated Offline run.
- Run the inference-endpoint tool against the live endpoint. The client records throughput in tokens per second and P90 time to first token.
- Write the disclosure files: system_desc.json for hardware and software configuration, and system_power.json for provisioned power, which feeds the power normalization metrics.
- Run local compliance checks with the submission tool to confirm data and accuracy targets are met.
- Submit with the same tool, which creates a submission ID and opens a peer-review pull request in a private MLCommons repository.

Faster results for submitters
Once submitted, results enter a continuous peer-review cycle and end up as a LoadStair chart that shows operating thresholds and capacity to enterprise buyers. MLCommons has also changed the timing rules. Provisional results can now be shared after an automatic check finishes, before peer review. Final verified results can be held under embargo when a company wants to time them to an event or launch.
Who it is for
Two groups have a stake. Providers get a way to show what their stack does when real customers push on it, which a chip-level score can’t capture. Buyers get the same evidence in one place, so they can compare vendors on measured behavior and not on marketing claims.
The agentic suite matters here. Multi-turn agent workloads stress a system differently from one-shot prompts, because context grows with every turn and batching gets harder. Having a fixed number of measurement points for those runs, 7 in all, keeps results comparable across submitters.
Power is part of the picture too. Each submission reports provisioned system power, and v1.0 uses it to normalize results, so a large cluster and a small one can be compared on the same footing.
What to expect
The test suite uses publicly available AI models, so results are open and reproducible. Whether it changes buying decisions depends on adoption. A benchmark is only as useful as the number of providers that publish on it, and early cohorts will show which neoclouds and hardware vendors are willing to expose their performance cliffs.
For buyers, the practical value is simple. A measured frontier across load levels answers a question that peak scores can’t: what happens to response time and throughput when real traffic arrives. MLCommons points first-time submitters to a dedicated guide, and further test suites are listed on its benchmarks page.
Source: MLCommons