A developer with no Verilog experience used coding agents to build an FPGA inference chip design that runs a language model at 213 tokens per second. The project started at 6.43 tokens per second, the speed of the first full-model, bit-exact build, and finished 33 times faster after a few days of mostly unattended agent work.
The bill came to about $10,000 in Claude and OpenAI tokens. The agent wrote roughly 3 million lines across the experiment history and parsed 30 million lines of output files while running hundreds of experiments.
How the experiment started
The developer had never worked with FPGAs, the reprogrammable chips whose logic and routing are set by the configuration loaded onto them. Curiosity about AI inference on that hardware led to AWS F2 instances, which carry an FPGA with 16 GB of high-bandwidth memory. Renting one was easier than buying a board, and Codex handled the reservation.
The model came from the founders of DeepGrove, who were working on an architecture with ternary weights. It suits memory-limited hardware. The code was mostly C++ with some Verilog, and the agent ran in the cloud through Conductor for several days.
The prompts were short. Many amounted to a pep talk, such as asking for a 2x speedup. A few carried real ML knowledge, like suggesting speculative decoding or asking why a step ran on the CPU instead of the chip.

How the FPGA inference chip got 33 times faster
No single trick did most of the work. Each step added roughly 1.5x or less, and the gains compounded over 14 optimization steps.
- Cutting host software overhead, from about 10 ms per command to 0.9 ms, took the chip from 6.43 to 8.07 tokens per second.
- An attention engine with 128 lanes, which fetches the KV cache in bursts, lifted it to 14.02.
- A parallel router for the mixture-of-experts layers reached 20.53, and a glue engine for norms, RoPE and quantization reached 30.24.
- A new pipeline with a fresh sequencer jumped to 49.21 after the agent fixed two hangs that only showed up on real hardware.
Later gains for the FPGA inference chip came from exact speculative decoding, where the host drafts tokens and the FPGA checks several at once. Router weights had been crossing between dies at a sixth of the intended speed, and fixing that helped too. and lossless 5-trit weight packing cut weight bytes by 20%. The final step, a head loader that reads two memory banks at once, lifted throughput 17.3% to 213.13 tokens per second.
That last build runs at 225 MHz for the kernel and 425 MHz for memory. The agent chose a clock one step lower after a soak test turned up random faults at 231.25 MHz.
What the human advice changed
At around 10 tokens per second, the DeepGrove founders sent a prompt summarizing lessons from their own earlier FPGA attempt. They had tried a few months before and found the models weren’t good enough yet. The developer pasted it into the session as the most significant guidance of the whole run.
Afterward the agent reviewed its logs and reported which ideas it had used. Within 70 seconds of receiving the prompt, it had split the advice into ten ideas to adopt and five to reject. The big design ideas, such as streaming weights straight into compute lanes, were already in its plan. The five it rejected were numerics shortcuts like lower-precision activations, a deferred norm and an 8-bit KV cache, because each changes output bits and the project required bit-exact results.
The process advice mattered more. The agent killed a simulation that had been running for roughly a day, plus two 250 MHz builds. In their place it adopted a quick cycle of C tests, synthesis and a logic-depth check, saving place-and-route for each integration. That got hardware results sooner. The agent also flagged a cost it couldn’t prove: two hangs visible only in RTL simulation made it onto the board a day later, and the large integration then lost 38 hours.
While checking the memory suggestions, it also came across a reference design from AWS that hits 426 GB/s over all 32 channels, and built the memory subsystem on that.
Where humans still fit
The developer’s conclusion is measured. For most of the design ideas it is unclear whether the notes helped, since the agent came up with similar designs on its own many cycles later. People earned their place by setting how fast to iterate and what counts as correct, while the agents handled the exploration.
The comparison against other setups comes with caveats. The same optimization loop wasn’t run on every alternative, so the developer assumes more performance is left on the table. The GPU baselines also didn’t meet the same bit-exact bar, though they came close in practice.
What to watch next
The FPGA inference chip is one project on one model, so it doesn’t prove agents can design chips in general. It does lower the cost of trying. A founder with a large token budget and a clear correctness rule can now test hardware ideas that once needed a specialist team. The open question is how the approach holds up on models that don’t fit in 16 GB of memory.
Source: ankitg.me