A small trained image model beat two frontier LLMs at skin cancer diagnosis in one test

Dr Magnus Lynch, a consultant dermatologist at Guy's Hospital, gave 200 biopsy-proven skin lesions to GPT-6 Astra, Claude Fable 5.1 and Inception v3, an image model Google first published in 2015 that he fine-tuned for about twenty minutes on 68,011 public skin images, with the test lesions removed. The small model named the right diagnosis out of six options for 125 lesions (62%), against 84 and 85 (42%) for the two LLMs. He lists the limits himself: 200 lesions and 30 melanomas are small numbers, each LLM case was run once, and today's regulated skin cancer AI products perform considerably better than this baseline. The prompts, LLM replies and scoring code are public. On Hacker News (194 comments, 37 top-level), at least 12 top-level commenters called the result unsurprising or obvious, since a model trained for one narrow task should beat a general one. At least 4 questioned the test: the model was tuned on images from the same archive as the test set, the "10-year-old" label skips the fresh fine-tuning, and the LLMs gave one-shot answers instead of working with tools. One of them called it "benchmaxxing"; 5 of the 7 direct replies said tuning on in-domain data is normal practice, and one reply further down sided with him. The longest side thread was about using an LLM as a classifier at all: some said a prompt is faster when you have little labelled data, others said to use the LLM to help build a proper classifier. No consensus on how fair the test was. Sorted take: this is one doctor's small test, but the lesson travels. For a narrow job like sorting items into fixed buckets, a small model trained on your own labelled examples can beat a chat model.