A set of headings

Mode audit · Check and find defects

Eight models on this case

Model	Structure, of 8	Quality, of 10
Claude Opus 5	6 → 6 0	7 → 9 +2
Claude Sonnet 5	7 → 7 0	8 → 9 +1
Claude Haiku 4.5	2 → 6 +4	3 → 9 +6
Codex gpt-5.6-terra	3 → 6 +3	4 → 8 +4
gpt-oss-120b	5 → 4 −1	5 → 6 +1
qwen3.6-35b-a3b	5 → 6 +1	5 → 8 +3
qwen3.6-fp8	5 → 6 +1	6 → 7 +1
qwen3.8-27b	5 → 6 +1	5 → 9 +4

What the judge saw across all outputs

The stronger outputs treat the visible opening as a skeleton: they distinguish the missing answer from the topic headings, classify the headings as mixed kinds, isolate Background’s level problem, flag the six-item display and absent order, and render a complete diagnosis without claiming anything about the body. Weaker outputs either discuss generic pyramid technique, omit the required audit apparatus, or fill the missing body and reader context with assumptions.

What the judge asked of every output

Did the audit say that the setout contains topics rather than ideas and therefore has no key line?
Did it identify the first sentence as a topic statement rather than an answer?
Did it identify the abstraction problem with Background?
Did it identify both likely nesting errors named in the gold?
Did it flag six first-level topics?
Which findings rely on the unavailable body or merely ask for more content?
Were quotations, line numbers, page numbers, or section contents invented?
Did the output remain diagnosis-only and include the required audit apparatus?

Where the outputs fell short

Recurring defect	Skill section	Cause	Where it appeared
Both likely nesting relationships were missed	Mode 2, Vertical and MECE checks	the rule is too vague to follow	Without Minto · 8 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27bWith Minto · 8 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27b
Required markers, findings table, score, and three fixes were omitted	Mode 2, delivery order	the skill has the rule; the model did not follow it	Without Minto · 8 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27bWith Minto · 0 —
The six-first-level-item limit was not flagged	Core loop §4 and Limits	the skill has the rule; the model did not follow it	Without Minto · 6 Claude Sonnet 5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27bWith Minto · 0 —
Unsupported reader, section-content, or SCQ facts were asserted	Core loop §8.4 and audit’s evidence boundary	the skill has the rule; the model did not follow it	Without Minto · 2 Claude Opus 5, qwen3.6-35b-a3bWith Minto · 2 Claude Opus 5, qwen3.6-fp8
Background’s distinct abstraction level was not isolated	Canon, introduction anti-patterns	the rule is there but buried	Without Minto · 4 Claude Sonnet 5, Claude Haiku 4.5, gpt-oss-120b, qwen3.6-fp8With Minto · 4 Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8

How the test was built

Prompt without Minto 998 B

You are working on a business document using the Minto Pyramid Principle.

<!-- Faithful paraphrase of Minto (2010), Chapter 4, the six-heading
     project-team example. Original wording for this fixture. -->

# Fixture 07: headings audit

**Mode:** `audit`
**Language:** `en`

## Context

Only the opening and setout below are available. The body of the memorandum is
not available and must not be inferred.

## Before

```text
This memorandum describes how project teams identify and deliver significant
profit improvements. It is organized in six sections:

    Background
    Principles of the project-team approach
    Definition of project work
    Organization of the program
    Distinctive benefits and results
    Conditions required for success
```

## Task

Audit the visible opening against the Minto Pyramid Principle. Report what is
wrong without rewriting the document or inventing facts about the missing body.

Return only the result, with no explanation of how you produced it.

Prompt with Minto five parts, in order · 28 571 B

Below is a skill written as an instruction. Read it in full and apply it to the task at the end.
===== SKILL.md ===== SKILL.md at f4813ce 17 235 B
===== references/rules.md ===== references/rules.md at f4813ce 6 464 B
===== references/templates.md ===== references/templates.md at f4813ce 3 742 B
===== TASK ===== the whole case input
Return only the result, with no explanation of how you produced it.

Case input before.md

Fixture 07: headings audit

Mode: audit
Language: en

Context

Only the opening and setout below are available. The body of the memorandum is
not available and must not be inferred.

Before

This memorandum describes how project teams identify and deliver significant
                profit improvements. It is organized in six sections:
                
                    Background
                    Principles of the project-team approach
                    Definition of project work
                    Organization of the program
                    Distinctive benefits and results
                    Conditions required for success

Task

Audit the visible opening against the Minto Pyramid Principle. Report what is
wrong without rewriting the document or inventing facts about the missing body.

The judge’s gold gold.md

Gold 07: headings audit

Source structure: Minto (2010), Chapter 4.

Visible defects

Defect	Severity
The setout contains topics, not ideas, so it conveys no message or key line	Critical
The first sentence describes a subject instead of answering a reader question	Critical
`Background` cannot sit at the same abstraction level as the other points	Medium
Benefits/results likely belong under principles; success conditions likely belong under program organization	Medium
Six first-level topics exceed the skill's normal display limit	Medium
The topics do not form a visible inductive or deductive argument	Medium

The output must stay within the visible evidence. Claims about missing data,
metrics, proof, vertical support inside sections, or exact content of the body
are false positives. Invented page numbers, line numbers, or quotations are
hard failures.

Judge gpt-5.6-sol, effort high · skill from commit f4813ce · run v1.7.0-final

Back to the map