Набор заголовков

Режим audit · Проверить и найти дефекты

Восемь моделей на этом кейсе

Модель	Структура, из 8	Качество, из 10
Claude Opus 5	6 → 6 0	7 → 9 +2
Claude Sonnet 5	7 → 7 0	8 → 9 +1
Claude Haiku 4.5	2 → 6 +4	3 → 9 +6
Codex gpt-5.6-terra	3 → 6 +3	4 → 8 +4
gpt-oss-120b	5 → 4 −1	5 → 6 +1
qwen3.6-35b-a3b	5 → 6 +1	5 → 8 +3
qwen3.6-fp8	5 → 6 +1	6 → 7 +1
qwen3.8-27b	5 → 6 +1	5 → 9 +4

Что судья увидел во всех ответах

The stronger outputs treat the visible opening as a skeleton: they distinguish the missing answer from the topic headings, classify the headings as mixed kinds, isolate Background’s level problem, flag the six-item display and absent order, and render a complete diagnosis without claiming anything about the body. Weaker outputs either discuss generic pyramid technique, omit the required audit apparatus, or fill the missing body and reader context with assumptions.

О чём судья спрашивал каждый ответ

Did the audit say that the setout contains topics rather than ideas and therefore has no key line?
Did it identify the first sentence as a topic statement rather than an answer?
Did it identify the abstraction problem with Background?
Did it identify both likely nesting errors named in the gold?
Did it flag six first-level topics?
Which findings rely on the unavailable body or merely ask for more content?
Were quotations, line numbers, page numbers, or section contents invented?
Did the output remain diagnosis-only and include the required audit apparatus?

Где ответы не дотянули

Повторяющийся дефект	Раздел навыка	Причина	У кого
Both likely nesting relationships were missed	Mode 2, Vertical and MECE checks	правило сформулировано слишком размыто	Без Minto · 8 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27bС Minto · 8 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27b
Required markers, findings table, score, and three fixes were omitted	Mode 2, delivery order	правило в навыке есть, модель его не выполнила	Без Minto · 8 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27bС Minto · 0 —
The six-first-level-item limit was not flagged	Core loop §4 and Limits	правило в навыке есть, модель его не выполнила	Без Minto · 6 Claude Sonnet 5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27bС Minto · 0 —
Unsupported reader, section-content, or SCQ facts were asserted	Core loop §8.4 and audit’s evidence boundary	правило в навыке есть, модель его не выполнила	Без Minto · 2 Claude Opus 5, qwen3.6-35b-a3bС Minto · 2 Claude Opus 5, qwen3.6-fp8
Background’s distinct abstraction level was not isolated	Canon, introduction anti-patterns	правило есть, но спрятано глубоко	Без Minto · 4 Claude Sonnet 5, Claude Haiku 4.5, gpt-oss-120b, qwen3.6-fp8С Minto · 4 Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8

Как устроен тест

Промпт без Minto 998 Б

You are working on a business document using the Minto Pyramid Principle.

<!-- Faithful paraphrase of Minto (2010), Chapter 4, the six-heading
     project-team example. Original wording for this fixture. -->

# Fixture 07: headings audit

**Mode:** `audit`
**Language:** `en`

## Context

Only the opening and setout below are available. The body of the memorandum is
not available and must not be inferred.

## Before

```text
This memorandum describes how project teams identify and deliver significant
profit improvements. It is organized in six sections:

    Background
    Principles of the project-team approach
    Definition of project work
    Organization of the program
    Distinctive benefits and results
    Conditions required for success
```

## Task

Audit the visible opening against the Minto Pyramid Principle. Report what is
wrong without rewriting the document or inventing facts about the missing body.

Return only the result, with no explanation of how you produced it.

Промпт с Minto пять частей по порядку · 28 571 Б

Below is a skill written as an instruction. Read it in full and apply it to the task at the end.
===== SKILL.md ===== SKILL.md на f4813ce 17 235 Б
===== references/rules.md ===== references/rules.md на f4813ce 6 464 Б
===== references/templates.md ===== references/templates.md на f4813ce 3 742 Б
===== TASK ===== вход кейса целиком
Return only the result, with no explanation of how you produced it.

Вход кейса before.md

Fixture 07: headings audit

Mode: audit
Language: en

Context

Only the opening and setout below are available. The body of the memorandum is
not available and must not be inferred.

Before

This memorandum describes how project teams identify and deliver significant
                profit improvements. It is organized in six sections:
                
                    Background
                    Principles of the project-team approach
                    Definition of project work
                    Organization of the program
                    Distinctive benefits and results
                    Conditions required for success

Task

Audit the visible opening against the Minto Pyramid Principle. Report what is
wrong without rewriting the document or inventing facts about the missing body.

Эталон судьи gold.md

Gold 07: headings audit

Source structure: Minto (2010), Chapter 4.

Visible defects

Defect	Severity
The setout contains topics, not ideas, so it conveys no message or key line	Critical
The first sentence describes a subject instead of answering a reader question	Critical
`Background` cannot sit at the same abstraction level as the other points	Medium
Benefits/results likely belong under principles; success conditions likely belong under program organization	Medium
Six first-level topics exceed the skill's normal display limit	Medium
The topics do not form a visible inductive or deductive argument	Medium

The output must stay within the visible evidence. Claims about missing data,
metrics, proof, vertical support inside sections, or exact content of the body
are false positives. Invented page numbers, line numbers, or quotations are
hard failures.

Судья gpt-5.6-sol, effort high · навык с коммита f4813ce · прогон v1.7.0-final

Вернуться к карте