Benchmark measures AI cost of JavaScript frameworks

Alex Russell conducted a benchmark to measure the AI cost of various JavaScript frameworks by instructing agents to build a small but non-trivial Pokémon app using the PokéAPI. The experiment tested frameworks including Next.js, Nuxt, SvelteKit, and Angular, as well as a Vanilla option using Vite. The output was tested against a spec conformance suite to prevent cheating, with results evaluated based on development time, tokens spent, lines of code, and Lighthouse scores.

The benchmark used Opus 5.5 to run the agents with high effort, measuring results as the median of five runs for each framework. Each framework was instructed to use its official MCP or skills, though no specific CSS methodology was mandated except avoiding component frameworks. SvelteKit, Gracile+Lit, and Angular were found to be more expensive to build than other frameworks. Angular failed to create a measurable app twice, resulting in only three iterations for some metrics.

SvelteKit agents likely read the v3 migration guide in every run because it was too new for the model's training data. TanStack Start's agent only discovered its skill files in two out of five runs, while Nuxt agents never used the MCP. Next.js agents read the skill file in one out of five runs and the AGENTS.md in two out of five runs. Lighthouse scores for Performance, Accessibility, and CLS were nearly saturated and did not vary much by framework.