Mercor: AI benchmarks miss most of the work people actually do

Mercor mapped 29 agentic AI benchmarks onto the US government's O*NET list of 923 occupations and 18,797 tasks. Only 208 occupations have a task covered by at least 5 benchmark prompts, and Mercor estimates 90% of US working hours go to tasks with little or no benchmark coverage. Coverage leans towards large, well-paid, AI-exposed jobs: lawyers' tasks are heavily covered, while legal secretaries' and court clerks' tasks barely are. Mercor calls the results provisional, says a longer benchmark list would raise coverage, and notes its own APEX benchmarks are part of the list. On Hacker News (324 comments) there is no consensus. One camp agrees and says AI isn't touching the real bottlenecks, such as stakeholder disagreements and work with no quick feedback; one developer says he codes 20 times faster but now spends that time on presentations, Jira and QA. Another camp attacks the method: "covered" can mean as few as 13 prompts, hours should be weighted by economic value, and one practitioner says benchmarks favour the lowest-value tasks. Many replies are jokes. Sorted take: Don't choose a model for finance or ops work from a leaderboard; test it on ten of your own real tasks.