Could an AI agent figure out which skill to use just by trying things, instead of trusting whichever instructions sound closest to the task?

This explores whether an agent could find useful skills by testing them and keeping what works, rather than choosing whichever skill description sounds most similar to the task. The collection has no head-to-head study of the two approaches. It does make a strong case that semantic similarity is the wrong measure of usefulness, and it shows empirical testing working well one step over: deciding how a skill should be written rather than which skill to pick.

Start with why similarity falls short. One recurring diagnosis in retrieval research is that embeddings measure association, not relevance. Two texts can sit close together in embedding space without one being useful for the other's task, and there are mathematical limits on what a fixed-size embedding can represent at all Where do retrieval systems fail and why?. The memory literature gives a sharper warning. Memories that were recorded accurately and are genuinely relevant to the task still made LLM reasoning worse, and every memory framework tested scored below a no-memory baseline Can relevant memories actually harm LLM reasoning?. If relevant context can hurt, then "this skill looks relevant" doesn't tell you "this skill will help." Only trying it does.

The clearest case for testing is SkillOpt, which treats a skill document like model weights. An optimizer proposes edits to the text, and each edit is accepted only if it improves performance on held-out validation tasks Can skill documents be optimized like neural network weights?. That is empirical testing used as a filter on skill content, and it adds no cost when the agent runs. A similar logic shows up in training: tree search can stand in for human judgment by simply recording which solution paths succeed Can tree search replace human feedback in LLM training?. Across these examples, outcome signals keep replacing judgments about what looks right.

The less obvious finding is that retrieval may not be the main bottleneck. In compositional skill retrieval, the bigger problem is how the task gets broken into steps. Standard LLM decomposition matches only about a third of the steps it should, and fixing the step count recovers most of the lost performance What blocks skill retrieval in task decomposition?. If the agent splits a task into the wrong pieces, neither similarity search nor testing will find the right skill, because it is searching for the wrong things. Other work suggests the fix may be to change how you search rather than drop search entirely: agents that grep raw text beat embedding-based retrieval on precise queries Can direct corpus search beat embedding-based retrieval?, and routing a query to the right kind of knowledge structure beats one uniform retrieval method Can routing queries to task-matched structures improve RAG reasoning?.

The likely answer, then, is that testing doesn't replace retrieval but checks it. Retrieval proposes a few candidates cheaply, and testing confirms which ones actually help, the way SkillOpt's validation step gates every edit. Testing every skill in a large library would be expensive, and the collection doesn't yet show anyone doing it for selection. The part you might not expect is that the biggest gain may come before either step, from breaking the task into the right pieces.