Why does pure-vision underperform when parsing semantics and action prediction mix?
This explores why vision-only models (especially GUI agents reading raw screenshots) stumble when a single forward pass has to do two jobs at once — figure out what things mean and decide what to do — rather than failing at either job alone.
This explores why vision-only models stumble when a single pass has to both interpret a screen and act on it. The clearest answer in the corpus is OmniParser's: GPT-4V doesn't fail because it can't see, it fails because it's forced to identify icon meanings and predict actions simultaneously from raw pixels, and that composite task is the bottleneck Why do vision-only GUI agents struggle with screen interpretation?. Pre-parse the screenshot into structured, described elements and the model's job collapses to just action prediction — performance jumps. The lesson isn't "vision is weak," it's that bundling semantics and control into one step overloads a shared capacity.
What is that shared capacity? Two notes point at attention as the real resource being fought over. Verbose chain-of-thought actually *degrades* fine-grained perception because it optimizes verbalization when the genuine bottleneck is where the model looks — visual attention allocation, not how much it reasons out loud Does verbose chain-of-thought actually help multimodal perception tasks?. The complementary finding makes attention itself the thing worth optimizing: treating attention distributions as the policy target beats token-level RL on visual reasoning, because "attention is where the actual decision happens" Can optimizing attention patterns improve multimodal RL better than optimizing tokens?. Read together with OmniParser, a picture emerges — semantics and action both compete for the same limited attention budget, and mixing them starves each.
The corpus's repeated fix is to *offload the semantics into text* so the model only has to act. SignRAG shows that describing an unknown image in natural language, then retrieving against a text index, bridges the visual-reference gap better than direct embedding similarity Can describing images in text improve zero-shot recognition?. OmniParser does the same move for screens. In both, language acts as a relief valve: once meaning is named in text, the remaining task is narrow enough to do well.
There's a deeper current here worth surfacing, because it cuts the other way. Some notes argue text is a *lossy* abstraction — it strips physics, geometry, and causality, producing predictable failures in exactly the grounded reasoning a screen sometimes demands Are text-only language models fundamentally limited by abstraction?, and that meaning can't be reconstructed from form alone without shared intent Can language models learn meaning from text patterns alone?. So the parse-to-text trick buys focus at the cost of throwing away spatial and temporal detail. An interesting counter-direction: UI-JEPA learns user intent directly from unlabeled screen-recording video via temporal masking, keeping the visual-temporal signal instead of flattening it to a caption Can unlabeled UI video teach models what users intend?.
The thing you might not have known you wanted to know: "pure-vision underperforms" is rarely a perception failure. It's a *task-composition* failure — two cognitively distinct jobs sharing one attention budget — and the field has two opposing escapes. Decompose (parse semantics into text, leave action to the model) or re-target the optimizer at attention/temporal structure itself rather than the output tokens Can unlabeled UI video teach models what users intend?.
Sources 7 notes
OmniParser demonstrates that GPT-4V fails when forced to simultaneously identify icon meanings and predict actions from raw screenshots. Pre-parsing screenshots into structured semantic elements with descriptions lets the model focus solely on action prediction, removing the composite-task bottleneck.
Long rationales and text-token RL help reasoning but hurt fine-grained perception tasks because the actual bottleneck is visual attention allocation, not verbalization. Standard CoT optimization trains the wrong policy target.
Reinforced Attention Learning treats attention patterns as the primary policy target rather than token sequences. Direct optimization of information allocation shows stronger gains on visual reasoning than standard RLHF, because attention is where the actual decision happens.
SignRAG demonstrates that describing an unknown image via vision-language model, then retrieving known designs from a text-indexed database, eliminates the need for recognition model training. Natural-language description bridges the visual-reference gap better than direct embedding similarity.
Text strips the physics, geometry, and causality present in reality, forcing language models to manipulate symbols without grounding in their source dynamics. This creates predictable failure modes in physical, geometric, and causal reasoning that multimodal training could address.
Bender & Koller argue that meaning requires the relation between expressions and communicative intents. Since LLMs are trained only on form-to-form prediction with no access to shared attention or intent, they cannot reconstruct the meaning that grounds language.
UI-JEPA applies JEPA-style predictive masking to screen recordings, learning task-aware temporal representations that an LLM decoder can use to infer intent with minimal paired data. This trades the bottleneck of labeled video for abundant unlabeled streams.