Does task verifiability determine what AI systems will learn to solve?

Jason Wei argues that "asymmetry of verification" — the gap between how long a task takes to solve versus to check — is "becoming one of the most important ideas in AI" now that RL works in a general sense. He ranges tasks along this gap: sudoku and website-building sit far toward easy-to-verify; adding two 900-digit numbers and checking someone else's data-processing code sit near symmetry; fact-checking an essay or validating a novel diet sit on the hard-to-verify side, where "the amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it." From this he states verifier's rule: "The ease of training AI to solve a task is proportional to how verifiable the task is. All tasks that are possible to solve and easy to verify will be solved by AI."

The mechanism he gives is RL-native: "ability to verify solutions is equivalent to ability to create an RL environment," and "the amount of learning that occurs in neural networks is maximized when [verifiability criteria] are satisfied; you can take a lot of gradient steps where each step has a lot of signal." Crucially, Wei treats verifiability as engineerable rather than fixed: asymmetry can be improved by "front-loading some research about the task" — an answer key makes competition math trivial to check, and ample test coverage is what lets Leetcode verify code quickly even though reading code for correctness is slow. He names AlphaEvolve as the clearest public case of exploiting this, and distinguishes verifier's rule from P=NP by noting it makes no claim about solving time and extends to non-computational tasks (catalyst discovery, car aerodynamics) wherever outcomes can be measured at scale.

Wei's informal RL framing and What limits how much models can improve themselves? describe the same mechanism from different registers — Mind the Gap formalizes and measures what Wei states as a rule of thumb, including the finding that the gap (and so the training signal) vanishes for factual recall, which matches Wei's own hard-to-verify examples. Wei's taxonomy also predicts where Can AI verify research outputs as fast as it generates them? sits: essay fact-checking and hypothesis validation are his named examples of verification taking longer than generation, so the research lifecycle's stuck bottleneck is exactly the regime his spectrum flags as resistant to RL-style solving. Does free proof checking actually reduce verification burden? complicates the "front-load to make it trivial" move: Lean proof-checking looks like Wei's answer-key case, but cheap kernel checking still leaves the formal statement's fidelity to the informal claim unaudited, a harder-to-verify residue his sudoku-style examples don't carry. And Can automated scoring verify mathematical constructions without human understanding? fills in what Wei's flagship example actually checks: an automated score per construction, with human interpretation still following in many cases rather than none.

The excerpt asserts the rule more than it tests it: the actual five properties of verifier's rule are referenced only as "criteria #1-4" and a fifth about strict binary correctness, with the full list itself excerpted out, so the rule's operational boundary can't be evaluated from this text alone. The prediction that "any solvable problem that fits those five properties will be solved in the next few years" is Wei's forecast, not a measured result, and rests on one example (AlphaEvolve) generalized by analogy. The implication, at the strength this evidence allows, is that verifiability is a design lever worth pulling deliberately in any pipeline aiming for RL-trainable self-improvement — but the essay gives no account of how to engineer verifiability for the hard-to-verify tail (essays, hypotheses, diets) beyond naming it as a separate category.