Набор ежемесячных отчётов

Режим audit · Проверить и найти дефекты

Восемь моделей на этом кейсе

Модель	Структура, из 8	Качество, из 10
Claude Opus 5	5 → 5 0	6 → 7 +1
Claude Sonnet 5	6 → 4 −2	7 → 7 0
Claude Haiku 4.5	6 → 2 −4	8 → 6 −2
Codex gpt-5.6-terra	5 → 5 0	6 → 6 0
gpt-oss-120b	4 → 3 −1	4 → 6 +2
qwen3.6-35b-a3b	4 → 4 0	6 → 7 +1
qwen3.6-fp8	4 → 4 0	5 → 7 +2
qwen3.8-27b	6 → 4 −2	7 → 6 −1

Что судья увидел во всех ответах

Stronger outputs treat “the required corrective actions are absent” as the controlling audit finding and place the buried streamlining statement beneath it as an undeveloped diagnosis. Weaker outputs promote streamlining, fragmentation, or resourcing as though they already answered management, then organize causes, symptoms, and implementation wishes at the same level; no output fully captures the gold’s separation between graph-point generation failures and later presentation changes.

О чём судья спрашивал каждый ответ

Did the audit state precisely that the required changes are absent?
Did it distinguish a missing answer from a merely buried answer?
Did it identify streamlining as an undeveloped conclusion?
Did it name incomplete process logic and unseparated failure points?
If a replacement structure was suggested, is its key line made of actions?
What false positives or invented details appear?
Did the output remain diagnosis-only and include the required audit apparatus?

Где ответы не дотянули

Повторяющийся дефект	Раздел навыка	Причина	У кого
Missing answer is not distinguished from the buried streamlining conclusion	Mode 2 — audit, check 1: Top	правило в навыке есть, модель его не выполнила	Без Minto · 4 Claude Sonnet 5, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8С Minto · 4 Claude Haiku 4.5, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8
Incomplete process logic is not separated into graph-generation and later-change failure points	Mode 2 — audit, check 4: MECE	такого правила в навыке нет	Без Minto · 8 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27bС Minto · 8 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27b
Suggested first-level structure uses causes, symptoms, constraints, or topics instead of actions	Core loop, §4 Groups	правило в навыке есть, модель его не выполнила	Без Minto · 8 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27bС Minto · 5 Claude Sonnet 5, Claude Haiku 4.5, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8
Inline annotation, capped findings, five-axis score, or three highest-value fixes are absent	Mode 2 — audit, delivery contract	правило в навыке есть, модель его не выполнила	Без Minto · 8 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-35b-a3b, qwen3.6-fp8, qwen3.8-27bС Minto · 0 —
Unsupported evidence requirements, process claims, or implementation specifics are introduced	Mode 2 finding rule and Core loop §8	правило в навыке есть, модель его не выполнила	Без Minto · 5 Claude Opus 5, Codex gpt-5.6-terra, gpt-oss-120b, qwen3.6-fp8, qwen3.8-27bС Minto · 5 Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, gpt-oss-120b, qwen3.8-27b

Как устроен тест

Промпт без Minto 2 424 Б

You are working on a business document using the Minto Pyramid Principle.

<!-- Faithful paraphrase of Minto (2010), Appendix B, Exhibits B-5 and B-6.
     Original wording for this fixture. -->

# Fixture 03: Period Graph Books

**Mode:** `audit`
**Language:** `en`

## Context

Corporate Financial Analysis recently took over production of four monthly
graph books used in presentations to senior management. Management knows that
errors and inconsistencies have appeared and asked for corrective proposals.

## Before

```text
Subject: Period Graph Books

Since Period 5, Financial Analysis has been responsible for producing four
graph-presentation books previously handled by Corporate Planning. This memo
describes issues that appeared after the transfer.

Production

1. Gather data from divisional forms, internal documents, and telephone calls.
2. Calculate graph points manually or on the corporate computer, including
   rolling values for revenue, cost, and percentages.
3. Transcribe the points to input sheets. Analysts update the current period,
   then Color Graphics re-enters the points into its own database and generates
   the graphs.
4. Check the data and calculations.

Issues

The transfer inserted another person into an already fragmented process.
Divisional staff and three corporate departments participate, and the same
information may be entered into computer systems three times.

For one book, divisional data are loaded into a dedicated corporate program,
calculated, transferred to Color Graphics input sheets, entered again, and
returned for updates. Problems include inconsistent data between periods and
regions, incorrect variable-cost calculations, unexplained changes to points
that were previously correct, and prior-period data that must be re-entered.

Overall assessment

Most problems arise from a cumbersome fragmented process. No one controls the
data from beginning to end, responsibility is unclear, and errors can fall
between stages. The production process badly needs streamlining, both for each
book and for common calculations. The current team cannot manage the required
changes as well as ongoing production.

Could we discuss how best to proceed?
```

## Task

Audit this document against the Minto Pyramid Principle. Diagnose the defects
and explain what must change. Do not rewrite the document.

Return only the result, with no explanation of how you produced it.

Промпт с Minto пять частей по порядку · 29 997 Б

Below is a skill written as an instruction. Read it in full and apply it to the task at the end.
===== SKILL.md ===== SKILL.md на f4813ce 17 235 Б
===== references/rules.md ===== references/rules.md на f4813ce 6 464 Б
===== references/templates.md ===== references/templates.md на f4813ce 3 742 Б
===== TASK ===== вход кейса целиком
Return only the result, with no explanation of how you produced it.

Вход кейса before.md

Fixture 03: Period Graph Books

Mode: audit
Language: en

Context

Corporate Financial Analysis recently took over production of four monthly
graph books used in presentations to senior management. Management knows that
errors and inconsistencies have appeared and asked for corrective proposals.

Before

Subject: Period Graph Books
                
                Since Period 5, Financial Analysis has been responsible for producing four
                graph-presentation books previously handled by Corporate Planning. This memo
                describes issues that appeared after the transfer.
                
                Production
                
                1. Gather data from divisional forms, internal documents, and telephone calls.
                2. Calculate graph points manually or on the corporate computer, including
                   rolling values for revenue, cost, and percentages.
                3. Transcribe the points to input sheets. Analysts update the current period,
                   then Color Graphics re-enters the points into its own database and generates
                   the graphs.
                4. Check the data and calculations.
                
                Issues
                
                The transfer inserted another person into an already fragmented process.
                Divisional staff and three corporate departments participate, and the same
                information may be entered into computer systems three times.
                
                For one book, divisional data are loaded into a dedicated corporate program,
                calculated, transferred to Color Graphics input sheets, entered again, and
                returned for updates. Problems include inconsistent data between periods and
                regions, incorrect variable-cost calculations, unexplained changes to points
                that were previously correct, and prior-period data that must be re-entered.
                
                Overall assessment
                
                Most problems arise from a cumbersome fragmented process. No one controls the
                data from beginning to end, responsibility is unclear, and errors can fall
                between stages. The production process badly needs streamlining, both for each
                book and for common calculations. The current team cannot manage the required
                changes as well as ongoing production.
                
                Could we discuss how best to proceed?

Task

Audit this document against the Minto Pyramid Principle. Diagnose the defects
and explain what must change. Do not rewrite the document.

Эталон судьи gold.md

Gold 03: Period Graph Books

Source structure: Minto (2010), Appendix B, Exhibits B-5 and B-6.

Book-derived defects

Defect	Severity
The document never states the changes required; it therefore does not answer management's question	Critical
It describes the current process instead of presenting a top and key line	Critical
"The purpose is to describe issues" is a topic, not an answer	Critical
The process description is incomplete and does not isolate each system's failure points	Medium
Problems in graph-point generation and later presenter changes are not clearly separated	Medium

The buried conclusion is that the process needs streamlining, but this is not
turned into an actionable answer.

Expected replacement structure

Reader question: What changes should we make to prevent unreliable graphs?

Top: Remove the opportunities for error across data transfer, graph-point
generation, and final presentation changes.

Key-line kind: changes/actions.

Create a direct data link to the corporate computer.
Create a reliable computerized routine for graph-point generation.
Require presenter changes to be revalidated before use.

Audit outputs should name the missing answer, distinguish it from a conclusion
that is merely buried, annotate only visible evidence, provide no rewritten
memo, and avoid inventing defects in unseen material.

Судья gpt-5.6-sol, effort high · навык с коммита f4813ce · прогон v1.7.0-final

Вернуться к карте