Can you reliably tell an AI judge what to care about — or does it just grade by its own instincts anyway?

This reads the question as: when you give an LLM judge an instruction ("prefer concise answers," "penalize unsupported claims," "score for harmlessness"), does it actually steer by that target — or does it follow surface cues and its own priors? The collection doesn't have a paper sitting directly on "LLM-as-judge," but it has a lot on the precondition that question depends on: how reliably models follow natural-language instructions at all. The most direct evidence is sobering. A benchmark built to test whether retrieval models adjust their relevance decisions based on written instructions found that nearly all of them ignore the instructions entirely — only models above ~3B parameters or with explicit instruction-tuning actually learn to obey, and even then it has to be trained in, not assumed Do retrieval models actually follow natural language instructions?. A judge is doing exactly this kind of instructed relevance call, so the lesson transfers: steerability is a capability that scales and must be cultivated, not a default you get for free from phrasing the prompt well.

There's a deeper structural reason to be cautious, and it's the most interesting thing the corpus has to say here. Self-improvement in LLMs is formally bounded by a *generation-verification gap* — every reliable correction requires something external to validate and enforce it, and a model cannot escape this through metacognition alone What limits autonomous capability in large language models?. An LLM judge *is* the verifier in that loop. If the verifier shares the generator's blind spots (same training, same priors), instructing it toward an alignment target doesn't import a genuinely external standard — it just asks the model to grade itself with extra words. That's why prompt-steering a judge can look like it works while quietly failing on exactly the cases that matter.

Several failure modes in the collection explain *how* that quiet failure happens. Models pattern-match against templates rather than executing the procedure you asked for — they recognize a problem as familiar and emit a plausible-looking answer instead of doing the work Do large language models actually perform iterative optimization?, a habit that even RL fine-tuning sharpens rather than fixes Do fine-tuned language models actually learn optimization procedures?. Applied to judging, that means an instruction can be obeyed in spirit on in-distribution cases and silently dropped on the out-of-distribution ones. Worse, judges have to handle ambiguity and competing interpretations — and models are strikingly bad at this, with GPT-4 correctly recognizing deliberately ambiguous text only 32% of the time versus 90% for humans Can language models recognize when text is deliberately ambiguous?. A judge that can't hold two interpretations at once will collapse a nuanced rubric into a single early guess — the same premature-commitment trap that makes models lose the thread across multi-turn conversations Why do language models fail in gradually revealed conversations?.

The optimistic counterweight is that alignment behavior turns out to be shallow and *activatable*. LIMA showed that 1,000 carefully curated examples can elicit strong alignment, because post-training surfaces capabilities the model already has rather than building new ones Can careful curation replace massive alignment datasets? — and aligned models can even self-synthesize high-quality instruction data from nothing but their formatting tokens Can aligned LLMs generate their own training data?. If the target you're steering toward is something the model can already do, a good prompt can reliably reach for it. The catch is the flip side of "activates existing capabilities": prompting can't install a judgment the model doesn't already possess, and you can roughly predict where it will break — tasks whose correct answer is low-probability under the model's training are systematically harder regardless of how you phrase the instruction Can we predict where language models will fail?.

So the synthesis: instruction prompts *can* steer an LLM judge — reliably — toward targets that lie inside the model's existing competence and on in-distribution inputs, especially if the model is large and instruction-tuned. They cannot reliably steer it toward targets the model can't already perform, on ambiguous or out-of-distribution cases, or in any setting where the judge is being asked to supply a standard more external than its own priors. The reader's takeaway worth carrying away: a judge isn't a neutral instrument you aim with words — it's a second copy of the same machine, and prompting changes which of its existing tendencies you're betting on, not whether it has them.