{ "_spec_version": "1.0", "framework_id": "FRAMEWORK-SPEC-001", "name": "Underspecification Diagnostic", "version": "1.0.0", "created_date": "2026-09-04", "updated_date": "2026-09-04", "status": "production_ready", "creator": "Mike Goetz, RageDesigner", "synopsis": "When a generative model returns something bland, the reflex is to blame the model and reach for a different one. This framework says the output was correct and the question was not. A model asked an underspecified question returns the centre of its training distribution, because the centre is the answer with the lowest expected error across every possible person who could have asked. Generic is not a malfunction. It is the optimum for a question that never said who was asking or what would make the answer right.\n\nThe diagnostic is one question asked of the output rather than of the tool: what did the model not know that would have changed this? In practice the answer is one of three things, and they are the three this framework makes you write down. The deliverable is one plain text file per repeated task, carrying your standard, your context and your reasoning. The test is a side by side run, once cold and once with the file in front of the model. The comparison is the point rather than the output: if the constrained version is not better, the file is describing something you have not actually decided yet, and you found that out for the price of an hour instead of a quarter.", "origin_story": "Extracted in September 2026 from a week in which generated food photographs began appearing on real restaurant menus and were widely mocked for looking uniformly wrong. The available reading was that the tools are not good enough yet. The research underneath the story did not fit that reading: in a study of 60 matched image pairs, the AI images lost overall, and the 14 that beat the real photograph on realism mostly beat it on appetite too. Same generators, opposite outcomes. The tool was not the ceiling. The framework is the operational answer to what was, which turned out to be a question nobody had finished asking.", "classification": { "domain": "ai_collaboration", "category": "prompting_and_specification", "series": "SPEC", "tier": "leaf", "tags": [ "specification", "prompting", "generic_output", "standards", "ai_collaboration", "diagnostic" ], "triggers": [ "generic AI output", "why does AI output all look the same", "should I switch models", "my prompt is not working", "write down my standards", "should I generate this or shoot it" ] }, "coordinates_343": { "primary_meta_category": "Cognition", "primary_element_group": "judgment", "primary_coordinate": "Cognition.judgment.4.underspecification", "placement_confidence": "medium", "placement_rationale": "Placed on output rather than method. What this produces is a decision about what the operator withheld, plus a judgment call about whether specifying is the right move at all. The honest alternative is a systems placement, since the artifact is a written reusable spec. Judgment wins because the artifact is worthless when the underlying decision has not been made, which is the framework's own primary failure mode." }, "layer_1_principles_foundation": { "core_principles": [ { "principle": "Generic output is the correct answer to an underspecified question.", "description": "A generative model returns what minimises expected error across the distribution it was trained on. An unconstrained request has nothing narrowing that distribution, so the lowest-error answer is its centre. The model is not failing to try; it is succeeding at the question actually asked. Treating this as a defect sends you looking for a better tool, which is the one move that cannot help." }, { "principle": "The variable is what you withheld, not which model you picked.", "description": "Two operators with the same tool produce wildly different output. If capability were the binding constraint, the spread between users would be small. It is not." }, { "principle": "Almost always one of three things is missing: your standard, your context, your reasoning.", "description": "Standards are applied by reflex and so never written down. Context is invisible to you because it is ambient and invisible to the model because it was never in the data. Reasoning is rarest, because most people can state what they decided and very few can state how." }, { "principle": "Convergence ratchets, so the centre gets tighter over time.", "description": "Generated output is published, re-enters training data, and pulls the middle of the distribution harder toward itself. The cost of not specifying rises rather than falls." }, { "principle": "Some questions cannot be specified into range, and that is not a prompting failure.", "description": "Specification narrows toward an answer the distribution can reach. If the thing you want exists and is specifically yours, the model has never seen it and no prompt closes that gap. If it cannot be captured at all, generation is the best available route rather than a compromise, because nothing real is competing with it." } ] }, "layer_2_systematic_approach": { "methodology": "Gate on whether specification is the right move, recover the three missing inputs in writing, then prove the file earns its place by comparison rather than by belief.", "steps": [ { "step": 0, "name": "Gate: does the thing you want exist to be captured?", "description": "Three cases, three routes. Exists and looks like everyone else's: generate, and specification finishes the job. Exists and is specifically yours: capture it instead, because no prompt closes a gap in what the model was never shown. Cannot be captured at all: generation is the only route and the best case available. Only the first and third cases continue." }, { "step": 1, "name": "Pick the repeated task, not the interesting one", "description": "Choose a task you have run enough times to be right about. A task done twice yields a preference, not a standard." }, { "step": 2, "name": "Write the standard", "description": "What makes an output right or wrong, written so a competent stranger could apply it without asking you a question. The test is portability. 'Make it professional' fails, because two strangers would apply it differently. 'Every claim about the client's business traces to something they said on the intake call, and anything I inferred is marked as an inference' passes, because it can be checked." }, { "step": 3, "name": "Write the context", "description": "What is permanently true about your customer, constraint and market that the model has no route to. The three objections you always get. The regulator, the season, the price ceiling. Permanence is the filter." }, { "step": 4, "name": "Write the reasoning", "description": "Not what you decide, how you decide. The order you weigh things in, the tiebreaker, the condition under which you would do the opposite. Most people skip this layer, and it is what separates a style guide from a decision procedure." }, { "step": 5, "name": "Run the task twice and set the outputs side by side", "description": "Once cold, once with the file in front of the model. Do not judge the constrained run alone: an output read in isolation is judged against your memory of what you wanted, which is the thing you failed to write down." }, { "step": 6, "name": "Read the comparison as a diagnostic, not a result", "description": "Better means the file is real. Not better means the file describes something you have not actually decided, and that is the finding. Do not fix it by adding words; return to step 2." }, { "step": 7, "name": "Put reading the file into the run order", "description": "A specification nobody is required to consult reproduces the problem it was built to solve, quietly, because the file still exists and still looks like a control. The artifact is not the control; the requirement to read it is." } ] }, "layer_3_force_multipliers": { "primary_multipliers": [ { "name": "The file outlives the run", "mechanism": "A prompt improves one output and dies with it. A specification improves every future run without being rebuilt, so payback scales with how often the task recurs rather than with how well any single run went." }, { "name": "It separates expensive thinking from cheap execution", "mechanism": "Once standard, context and reasoning exist as text, a cheaper engine can execute against them at comparable quality, because the judgment now lives in the artifact rather than in the engine." }, { "name": "It is portable to people, not just models", "mechanism": "The same file that constrains a model onboards a contractor and settles an argument about what good looks like. Nothing about it is AI-specific, which is why it survives a change of tool." }, { "name": "A bounded self-contained spec retrieves well", "mechanism": "It is about one thing and states its own conditions of use, so it comes back as a complete procedure rather than as paragraphs that happen to mention the topic." } ] }, "layer_4_success_metrics": { "leading_indicators": [ "The constrained run is visibly better than the cold run on first comparison, without argument about which is better.", "You can state the standard in a sentence a stranger could apply, with no adjectives doing load-bearing work.", "The reasoning layer contains at least one tiebreaker and one condition for doing the opposite.", "The file fits on roughly one page and you did not have to invent anything to fill it." ], "lagging_indicators": [ "The file is reused on the next run without being rebuilt or argued with.", "Someone other than the author produces an acceptable output from it unaided.", "Output quality stops depending on which model or which day.", "The file gets edited when the work changes, rather than every time it runs." ], "failure_modes": [ { "mode": "Adjective standard", "signal": "The standard is a list of words like professional, clean or engaging that two people would apply differently.", "correction": "Rewrite as something checkable. If it will not become checkable, this is judgment you have not yet decomposed." }, { "mode": "Document creep", "signal": "The file grows past a page and stops being read before running.", "correction": "Split until one file governs one task, and apply the permanence filter to every context line." }, { "mode": "Different but not better", "signal": "The constrained output changed without improving.", "correction": "The standard encoded preference rather than judgment. Return to step 2." }, { "mode": "Silent drift", "signal": "You edit the file every single run.", "correction": "That is not specification, it is the surface underneath having moved. The file has quietly become the patch between the document and reality." }, { "mode": "The unread control", "signal": "The file exists, is correct, and is never opened.", "correction": "Step 7. This failure produces no error, which is why it is the most common one." } ], "red_flags_do_not_proceed": [ "The task has been done twice. Encoding an unformed standard makes an unfinished thing permanent and much harder to see later.", "You are specifying to avoid deciding. If the reasoning layer will not write, the decision has not been made.", "The output you want exists and is specifically yours. Capture it instead.", "You actually want the market average, which is a real and common case: benchmarking, checking convention, drafting a first pass you intend to argue with." ] }, "layer_5_implementation_guidance": { "entry_conditions": { "required": [ "A generative output missed, and the task is one you repeat.", "You can run the same task twice in one sitting to produce the comparison." ], "optimal": [ "You are about to switch models to fix a quality problem.", "You are about to hand a repeated task to another person or to an agent." ] }, "exit_conditions": [ "The file exists, the comparison favours it, and reading it is a step in the run order.", "A competent stranger produced an acceptable output from the file without asking a question.", "Or the comparison did not favour it, and you have correctly concluded the decision is not yet made." ], "edge_cases": [ { "case": "The market average is the goal", "guidance": "Do not run this. Asking for generic and receiving generic is a success." }, { "case": "Your own specificity is wrong", "guidance": "The dangerous case, and worse than generic. A confidently encoded bad assumption produces confidently wrong output, which does not look generic. It looks considered. Generic output announces itself; a wrong standard applied consistently announces nothing and can run for a year. Keep the cold run as a control and re-compare when the work changes." }, { "case": "The work is genuinely one-off", "guidance": "Writing the standard costs more than the task. Specify inline and build no artifact." } ], "time_investment": "About one hour for the first file on a task you already know well. The comparison run is ten minutes of it.", "prerequisites": [ "A task you have repeated enough times to be right about it.", "Access to run the same task twice, cold and constrained." ] }, "relationships": { "depends_on": [], "extends": [], "related_to": [ "FRAMEWORK-0330-001", "FRAMEWORK-ABP-001", "FRAMEWORK-ROUTE-001" ], "conflicts_with": [], "notes": "FRAMEWORK-0330-001 (Visual DNA) is this framework already specialised to image generation, so use it directly rather than re-deriving steps 2 through 4 for visual work. FRAMEWORK-ABP-001 (Anti-Barnum) is the mirror image and a necessary guard: this framework addresses output that is recognisably generic, Anti-Barnum addresses output that is generic while feeling specific to you. A specification can pass the comparison test here and still be Barnum." }, "evidence_base": { "description": "One measured result and one mechanism. The bound on what the study supports is stated, because overreaching from it would violate this framework's own principle.", "sources": [ { "source": "Diel et al., Scientific Reports, 2026. AI-generated food stimuli match real counterparts in perceived healthiness and calorie content but not in realism or willingness to eat.", "finding": "87 participants rated 60 pairs of AI-generated and real food images matched for calorie content and flavour. The AI images were rated significantly less realistic and drew lower willingness to eat overall. Among the 14 AI images rated more realistic than their real counterpart, 11 also drew higher willingness to eat.", "supports": "That generator capability is not the binding constraint, because the same generators produced both outcomes.", "does_not_support": "It does NOT establish specification as the cause. The study measured a relationship between perceived realism and willingness to eat. It never manipulated how any image was specified and cannot say why those 14 landed. It was also not a study of menus. Any claim that specification is why the 14 won is a fabricated mechanism.", "sourcing_note": "Abstract obtained from the publisher's Crossref deposit; the 14-of-60 detail comes from a secondary write-up rather than the abstract." }, { "source": "Alex Lisle, CTO of Reality Defender, in TechCrunch, 2026-09-03.", "finding": "Named convergence and distinguished it from model collapse: generated output is published, re-enters training data, and pulls the centre of the distribution tighter.", "supports": "The fourth principle, that the cost of not specifying rises over time.", "interest_disclosure": "Reality Defender sells AI detection, so Lisle is commenting on the category his company sells into." } ], "stated_interest": "This framework's central claim, that the model is not the variable and you are, is convenient for anyone selling a model, because it moves responsibility for bad output from seller to buyer. It is carried anyway because the mechanism is checkable independently of anyone's commercial interest, because the same principle says the default output of these tools is bad enough to cost a business its customers, and because step 0 tells a large class of users not to generate at all. A vendor's argument would not contain that step." }, "version_history": { "v1_0_2026_09_04": "First public release. Carries the three-input diagnostic (standard, context, reasoning), the step 0 capture gate that stops the framework being applied to work whose answer was never in the distribution, and the evidence bound that prevents the cited study being read as proof of a mechanism it never measured." } }