Work2Skill turns human desktop recordings into reusable skills for computer-use agents
Sorted AI13 OctResearchers introduce Work2Skill, a framework that captures how a person uses desktop software and turns it into reusable skills for computer-use agents. A tool called WorkRecorder records synchronised screen video, mouse and keyboard events and available accessibility-tree information. Using it, the team built WorkDemo-350, a dataset of 350 human GUI demonstrations screened for completeness and task success. An "agentic skill induction" method then inspects this evidence to extract workflow rules, step dependencies and completion checks. The paper was submitted to arXiv on 9 October 2026.
When the same four execution models were each given skills induced by different methods under shared settings, the authors report that Work2Skill scored highest. Mean score gains over each model's no-skill baseline were 4.67 to 10.54 points on MyPCBench Bounded Action and 3.11 to 7.50 points on a 286-task subset of OSWorld-Verified. The code and data are on GitHub. These are the authors' own results.
Why it matters: If you want agents to handle niche internal software, learning from a few recorded human runs is a cheaper path than hand-writing instructions, though the reported gains are modest and from the authors' own tests.
arXiv ↗AI-assisted, reviewed by Sorted AI
▲ 0
0 comments