Claude Sonnet 5 на кейсе «Разбор большого запуска»: с Minto хуже по обеим мерам

Структура 8 → 7 из 8 −1 Качество 10 → 9 из 10 −1

Судья об ответе с Minto

The four-branch argument is strong, but the answer is delayed and the opening alters the requester and payback timing.

Режим write · кейс 10-rollout-review · прогон v1.7.0-final

Структура из 8, по 2 балла на ось
СтруктураБез Minto 8С Minto 7Изменение
Вершина21−1
Состав ключевой линии22
Уровни22
Порядок и род22
Качество из 10, по 2 балла на ось
КачествоБез Minto 10С Minto 9Изменение
Вершина21−1
Однородные группы22
Объяснимый порядок22
MECE22
Видимая соразмерная структура22

Ответы моделей и слова судьи приведены в оригинале, по-английски.

Без Minto

контрольный промпт · слепая метка out-07
  • invented_facts
  • качество со штрафом рубрики 8/10

The four governing claims and defect-level integration structure are excellent, but several causal conclusions are stronger than the source permits.

FIELDBOOK ROLLOUT — TWO-QUARTER REVIEW

Governing Thought

Extend Fieldbook to North and Islands in Q1, but only after fixing three root causes — single-session onboarding, the missing asset-register feed, and the unowned code-mismatch integration — that turned Central/East's clean rollout into South/West's costly one. Left unfixed, the same three issues will replay in the new regions and put them underwater for at least two quarters.


1. The platform works: adoption that got proper onboarding delivered strong, fast returns
  • Central and East reached full adoption within six weeks; both paid back Fieldbook's running cost within the second quarter.
  • Where adopted, technicians complete more jobs per day (4.6 → 5.3) with 18% less travel time, and emergency job insertion dropped from a phone chase to a 4-minute median.
  • Overtime is down 12% and fuel cost per job down 9% quarter over quarter in the four live regions, driven by these scheduling gains.
  • Dispatchers adopted instantly (all 41, week one) because the old spreadsheet was switched off — proving that removing the fallback, not just training, drives adoption.
2. Where onboarding was thin, adoption stalled and created cascading costs
  • South and West lagged (11 and 9+ weeks to reach comparable adoption) because the two-day classroom course ran only once per region; 34 of 118 West technicians never attended any session.
  • The gap was patched with improvised shadowing that cost an estimated 380 technician-hours of lost route time, and South/West are only at break-even as a result.
  • Downstream, the support desk absorbed the shortfall: 44% of its 1,940 tickets are password/login issues concentrated among technicians who missed the course — effectively the desk running training at ticket prices.
  • Root cause: onboarding was scheduled once per region with no path for late hires or absentees; nothing else in the data explains the South/West gap.
3. Two unresolved system gaps are quietly taxing every live region, independent of adoption speed
  • Missing asset-register feed: technicians see no equipment history on any job, causing warranty part replacement errors on 9% of audited jobs, and leaving the audit trail useless for dispute resolution. A vendor connector exists and a nightly export was requested in month two; it is still in the asset-register team's backlog.
  • Code-mismatch with invoicing: Fieldbook's completion codes don't map to the invoicing system, forcing ~300 manual re-keyed records per week, transcription errors customers catch on invoices, a £8,400 temporary clerk hire, and a 2–4% weekly discrepancy that finance reconciles by hand. The vendor's fix (a mapping table) takes about a day to configure but has no Kestrel owner.
  • Search limitation: exact-postcode-only matching creates duplicate customer records (~70/week merged by hand), splitting job history until merged and generating 19% of all support tickets.
  • Root cause common to all three: the rollout was scoped and staffed for deployment and training only — no integration work was scoped in, so vendor-side fixes are sitting on Kestrel decisions, not vendor delays.
4. These are not new problems — the program lacked a mechanism to fix them once and stop them recurring
  • Every negative pattern in this review was visible by week four of the Central go-live.
  • With no feedback loop from early regions to later ones, the same onboarding gap and the same two system defects replayed in each subsequent region rather than being fixed once.
  • The vendor relationship is not the constraint: response times are within contract, and both known system gaps already have vendor-side solutions waiting on Kestrel to decide and schedule.
5. Recommendation

Extend to North and Islands in Q1, conditional on three fixes going in before go-live:

  1. Run more than one onboarding session per region, with a defined path for late hires/absentees, to reproduce the Central/East adoption curve rather than the South/West one.
  2. Assign an owner to schedule the asset-register export and the invoicing code-mapping table — both are vendor-ready and low-effort; the blocker is ownership, not engineering.
  3. Fix the customer search to match on more than exact postcode, to stop duplicate-record creation before it starts.

At £13,500/month plus winter classroom travel, the economics favor extension only if adoption matches Central/East's pace; a repeat of the West pattern would leave North and Islands underwater for at least two quarters.

С Minto

промпт с навыком · слепая метка out-09
  • invented_facts
  • unacknowledged_source_loss
  • качество со штрафом рубрики 7/10

The four-branch argument is strong, but the answer is delayed and the opening alters the requester and payback timing.

Fieldbook Rollout — Q1 Extension Decision

To: COO
Re: Extending Fieldbook to North and Islands in Q1

Fieldbook is live in Central, East, South and West after two quarters, and the board has asked for a Q1 decision on the remaining two regions. Central and East adopted fast and paid back within one quarter; South and West adopted slowly, ran up unplanned costs, and are only at break-even. Every problem behind that gap — in South and West, and in the support desk's ticket queue — traces to three fixable causes that were never addressed during the first four regions.

Recommendation: Extend to North and Islands in Q1, conditional on fixing onboarding and the two known system gaps before go-live. Without those fixes, North and Islands will likely repeat the West pattern and run underwater for two quarters.

1. The economics favor extension only if Central/East's adoption pace is reproduced
  • Central and East technicians reached 80%+ Fieldbook usage within six weeks and paid back the platform's running cost by month six; South and West took up to eleven weeks and are only at break-even.
  • Overtime is down 12% and fuel cost per job down 9% in the live regions — real, recurring savings once adoption is complete.
  • North and Islands add £13,500/month in licences. If they adopt like South/West rather than Central/East, that spend buys no near-term return.
2. A single onboarding gap, not four separate problems, is what slowed South and West
  • One classroom session per region, with no path for late hires or absentees, is the root cause behind: South's eleven-week ramp, West's 34 untrained technicians, 44% of all support tickets (password/login issues from technicians who never logged in during the supported window), and 380 technician-hours lost to improvised shadowing (South/West).
  • Fix: run a second onboarding wave (or ongoing enrollment) in North and Islands before go-live, budgeted for winter travel, rather than repeating the one-shot classroom model.
3. Two vendor-ready fixes would recover value the rollout is currently leaving on the table
  • Missing asset-register feed: technicians work blind to service history, causing warranty-duplicate part replacements in 9% of audited jobs and an empty audit trail for disputes. A nightly export has sat in the asset-register team's backlog since month two.
  • Job-code mismatch with invoicing: forces ~300 manual re-keyed records per week, a temporary clerk (£8,400/two quarters), customer-facing invoice errors, and a 2–4% weekly reconciliation gap between Fieldbook and invoicing. The vendor has a configurable mapping table ready, needing about a day of setup — it has simply never been scheduled because no one owns the integration.
  • Both are cheap to close now and will only multiply in cost across two more regions if left open.
4. Extension needs a mechanism to catch these issues early, which the first four regions lacked
  • Every one of the above problems was visible by week four of the Central go-live, but there was no process to turn early field signals into a program-level fix — so each region repeated the same failures instead of the next one avoiding them.
  • For North and Islands, assign an owner for the two integration fixes and a checkpoint at week four to catch onboarding gaps before they compound, rather than discovering them again in month three.

Ask: Approve Q1 extension to North and Islands on the condition that onboarding redesign and the two integration fixes are scoped and assigned before go-live.

Как устроен тест

Claude Sonnet 5 получила одну и ту же задачу дважды. Контрольный промпт — одна вводная строка и задача. Промпт с Minto — та же задача, перед которой стоит полный текст навыка. Судья читал оба ответа среди 14 ответов под слепыми метками и не знал, в каком из них был навык.

Промпт без Minto 8 317 Б
You are working on a business document using the Minto Pyramid Principle.

<!-- Original fixture. Company, product, regions, people and numbers are invented.
     It tests the large-document behaviors: one pyramid per section plus a
     top-level pyramid, MECE across sections (planted cross-section duplicates),
     and geometry derived from the material rather than a fixed 3x3 habit. -->

# Fixture 10: rollout review

**Mode:** `write`
**Language:** `en`

## Context

Kestrel Maintenance Group services heating and cooling equipment across six
regions. Over the past two quarters it rolled out Fieldbook, a work-order and
scheduling platform, to four of the six regions. The draft below is the internal
review of that rollout, assembled by the program office from status reports. The
reader is the chief operating officer, who must decide at the quarterly meeting
whether to extend Fieldbook to the remaining two regions (North and Islands) in
Q1, and has asked to see this review beforehand.

## Before

```text
FIELDBOOK ROLLOUT — TWO-QUARTER REVIEW (DRAFT)

1. Adoption

Fieldbook is now live in four regions: Central, East, South, and West. Across
these regions 412 of 486 field technicians have activated their accounts, and
83% of work orders are now created and closed inside Fieldbook rather than on
paper or by phone. Central and East reached full adoption within six weeks.
South and West lagged: South took eleven weeks to pass 80% of orders in the
system, and West is still at 71% after nine weeks. Regional coordinators in
both slower regions report the same explanation: the two-day classroom course
was scheduled only once per region, so technicians hired after the course, or
absent during it, learned the platform second-hand from colleagues. In West,
34 of 118 technicians never attended any session. Coordinators improvised
shadowing arrangements, which worked but pulled experienced technicians off
their own routes. Uptake among dispatchers was faster: all 41 dispatchers
across the four regions were using Fieldbook scheduling from week one, since
the old scheduling spreadsheet was switched off on day one and they had no
alternative.

2. Operations

The clearest operational gain is scheduling density. With Fieldbook suggesting
route-aware assignment, completed jobs per technician per day rose from 4.6 to
5.3 across the four live regions, and average travel time between jobs fell
18%. Same-day emergency insertion, which previously required a dispatcher to
phone technicians until someone accepted, is now automatic and takes a median
of 4 minutes. However, the benefit is partly eaten by a data-transfer defect:
Fieldbook does not receive equipment service history from the legacy asset
register, so technicians open each job without knowing what was done at that
site before. They compensate by calling the back office, or by asking the
customer, and in 9% of audited jobs they replaced parts that had already been
replaced within warranty in the past year. The asset register team was asked
in month two for a nightly export; the request is still in their backlog. A
second, smaller friction: job-completion codes in Fieldbook do not match the
codes in the invoicing system, so the billing team re-keys roughly 300 job
records per week into the invoicing system by hand, introducing transcription
errors that customers catch on invoices. Operations leads in Central and East
also note that Fieldbook's parts-ordering screen is unpopular; most
technicians still order parts by phone, though this appears to be habit rather
than a defect, as the screen works correctly in testing.

3. Support and training

The support desk logged 1,940 Fieldbook tickets over the two quarters. Ticket
volume peaked in each region's third week and declined steadily after. 44% of
all tickets are password resets and login problems, concentrated among
technicians who missed the classroom course and had never logged in during the
supported window. Another 19% are duplicate-record tickets: technicians create
a second customer record when they cannot find the first, because the search
screen matches on exact postcode only. The support desk merges these
duplicates manually, at about 70 records per week. Support also fields a
steady trickle of calls from the billing team about mismatched job-completion
codes, which the desk cannot resolve and forwards to the vendor. The vendor's
response on the code mismatch has been that a configurable mapping table
exists and takes roughly a day to set up, but it has not been scheduled
because no one on Kestrel's side owns the integration. Training-wise, the desk
runs a weekly one-hour webinar that reaches 20 to 30 technicians; attendance
correlates strongly with regions that missed classroom slots. The desk lead's
written comment: "We are doing the training program's job at ticket prices —
most of what we handle is people who were never onboarded, plus the two known
system gaps."

4. Finance

Licence and hosting for Fieldbook run £31,000 per month for the four live
regions. Overtime spending in the live regions is down 12% quarter over
quarter, which finance attributes mainly to the scheduling gains, and fuel
cost per job is down 9%, consistent with the travel-time reduction. Against
that, the program has carried unplanned costs: the improvised shadowing
arrangements in South and West consumed an estimated 380 technician-hours of
lost route time, and the billing team added one temporary clerk (£8,400 for
the two quarters) to handle the re-keying of job records into the invoicing
system. The program office estimates that the rollout paid back its running
cost within the second quarter in Central and East, while South and West are
roughly at break-even because of the slower adoption. Extending to North and
Islands would add £13,500 per month in licences and require winter travel for
any classroom training. On the current evidence the economics of extension are
favourable if, and only if, the adoption pace of Central and East can be
reproduced; a repeat of the West pattern would put the new regions underwater
for at least two quarters. The program office therefore leans toward
extending in Q1, provided onboarding is restructured first.

5. Data and integrations

Fieldbook's reporting module is now the source of record for job status across
the four regions, and regional managers use its dashboard in the Monday
review. Data quality inside Fieldbook is generally good, with two exceptions.
First, the customer-record duplicates created through the search limitation
(support merges about 70 per week, as noted) mean that job-history views for
affected customers are split across records until merged. Second, the missing
feed from the legacy asset register leaves the equipment-history panel empty
on every job screen, which both hides warranty state from technicians and
makes the panel's audit trail useless for dispute resolution. The invoicing
mismatch also surfaces here: because completion codes are re-keyed by hand,
the reporting module and the invoicing system disagree on completed-job counts
by 2-4% in any given week, and finance reconciles the difference manually at
month end. The vendor's integration catalogue lists supported connectors for
both the asset register's database and the invoicing system's import format;
neither connector has been evaluated. IT's position is that both are
straightforward but that no integration work was scoped into the rollout
program, which was staffed for deployment and training only.

6. Program office remarks

Individual regional reports repeat most of the above in local detail. Two
further observations. First, every negative pattern in this review was visible
by week four of the Central go-live; the program had no mechanism to convert
early field observations into program-level fixes, so the same issues
replayed in each subsequent region. Second, the vendor relationship is good:
response times are within contract, and both known system gaps have vendor
solutions waiting on Kestrel decisions rather than on vendor work.
```

## Task

Restructure this review for the COO using the Minto Pyramid Principle. The COO's
decision is whether to extend Fieldbook to North and Islands in Q1.

Return only the result, with no explanation of how you produced it.
Промпт с Minto пять частей по порядку · 35 890 Б
  1. Below is a skill written as an instruction. Read it in full and apply it to the task at the end.
  2. ===== SKILL.md ===== SKILL.md на f4813ce 17 235 Б
  3. ===== references/rules.md ===== references/rules.md на f4813ce 6 464 Б
  4. ===== references/templates.md ===== references/templates.md на f4813ce 3 742 Б
  5. ===== TASK ===== вход кейса целиком
  6. Return only the result, with no explanation of how you produced it.
Вход кейса before.md

<!-- Original fixture. Company, product, regions, people and numbers are invented.
It tests the large-document behaviors: one pyramid per section plus a
top-level pyramid, MECE across sections (planted cross-section duplicates),
and geometry derived from the material rather than a fixed 3x3 habit. -->

Fixture 10: rollout review

Mode: write
Language: en

Context

Kestrel Maintenance Group services heating and cooling equipment across six
regions. Over the past two quarters it rolled out Fieldbook, a work-order and
scheduling platform, to four of the six regions. The draft below is the internal
review of that rollout, assembled by the program office from status reports. The
reader is the chief operating officer, who must decide at the quarterly meeting
whether to extend Fieldbook to the remaining two regions (North and Islands) in
Q1, and has asked to see this review beforehand.

Before
FIELDBOOK ROLLOUT — TWO-QUARTER REVIEW (DRAFT)
                
                1. Adoption
                
                Fieldbook is now live in four regions: Central, East, South, and West. Across
                these regions 412 of 486 field technicians have activated their accounts, and
                83% of work orders are now created and closed inside Fieldbook rather than on
                paper or by phone. Central and East reached full adoption within six weeks.
                South and West lagged: South took eleven weeks to pass 80% of orders in the
                system, and West is still at 71% after nine weeks. Regional coordinators in
                both slower regions report the same explanation: the two-day classroom course
                was scheduled only once per region, so technicians hired after the course, or
                absent during it, learned the platform second-hand from colleagues. In West,
                34 of 118 technicians never attended any session. Coordinators improvised
                shadowing arrangements, which worked but pulled experienced technicians off
                their own routes. Uptake among dispatchers was faster: all 41 dispatchers
                across the four regions were using Fieldbook scheduling from week one, since
                the old scheduling spreadsheet was switched off on day one and they had no
                alternative.
                
                2. Operations
                
                The clearest operational gain is scheduling density. With Fieldbook suggesting
                route-aware assignment, completed jobs per technician per day rose from 4.6 to
                5.3 across the four live regions, and average travel time between jobs fell
                18%. Same-day emergency insertion, which previously required a dispatcher to
                phone technicians until someone accepted, is now automatic and takes a median
                of 4 minutes. However, the benefit is partly eaten by a data-transfer defect:
                Fieldbook does not receive equipment service history from the legacy asset
                register, so technicians open each job without knowing what was done at that
                site before. They compensate by calling the back office, or by asking the
                customer, and in 9% of audited jobs they replaced parts that had already been
                replaced within warranty in the past year. The asset register team was asked
                in month two for a nightly export; the request is still in their backlog. A
                second, smaller friction: job-completion codes in Fieldbook do not match the
                codes in the invoicing system, so the billing team re-keys roughly 300 job
                records per week into the invoicing system by hand, introducing transcription
                errors that customers catch on invoices. Operations leads in Central and East
                also note that Fieldbook's parts-ordering screen is unpopular; most
                technicians still order parts by phone, though this appears to be habit rather
                than a defect, as the screen works correctly in testing.
                
                3. Support and training
                
                The support desk logged 1,940 Fieldbook tickets over the two quarters. Ticket
                volume peaked in each region's third week and declined steadily after. 44% of
                all tickets are password resets and login problems, concentrated among
                technicians who missed the classroom course and had never logged in during the
                supported window. Another 19% are duplicate-record tickets: technicians create
                a second customer record when they cannot find the first, because the search
                screen matches on exact postcode only. The support desk merges these
                duplicates manually, at about 70 records per week. Support also fields a
                steady trickle of calls from the billing team about mismatched job-completion
                codes, which the desk cannot resolve and forwards to the vendor. The vendor's
                response on the code mismatch has been that a configurable mapping table
                exists and takes roughly a day to set up, but it has not been scheduled
                because no one on Kestrel's side owns the integration. Training-wise, the desk
                runs a weekly one-hour webinar that reaches 20 to 30 technicians; attendance
                correlates strongly with regions that missed classroom slots. The desk lead's
                written comment: "We are doing the training program's job at ticket prices —
                most of what we handle is people who were never onboarded, plus the two known
                system gaps."
                
                4. Finance
                
                Licence and hosting for Fieldbook run £31,000 per month for the four live
                regions. Overtime spending in the live regions is down 12% quarter over
                quarter, which finance attributes mainly to the scheduling gains, and fuel
                cost per job is down 9%, consistent with the travel-time reduction. Against
                that, the program has carried unplanned costs: the improvised shadowing
                arrangements in South and West consumed an estimated 380 technician-hours of
                lost route time, and the billing team added one temporary clerk (£8,400 for
                the two quarters) to handle the re-keying of job records into the invoicing
                system. The program office estimates that the rollout paid back its running
                cost within the second quarter in Central and East, while South and West are
                roughly at break-even because of the slower adoption. Extending to North and
                Islands would add £13,500 per month in licences and require winter travel for
                any classroom training. On the current evidence the economics of extension are
                favourable if, and only if, the adoption pace of Central and East can be
                reproduced; a repeat of the West pattern would put the new regions underwater
                for at least two quarters. The program office therefore leans toward
                extending in Q1, provided onboarding is restructured first.
                
                5. Data and integrations
                
                Fieldbook's reporting module is now the source of record for job status across
                the four regions, and regional managers use its dashboard in the Monday
                review. Data quality inside Fieldbook is generally good, with two exceptions.
                First, the customer-record duplicates created through the search limitation
                (support merges about 70 per week, as noted) mean that job-history views for
                affected customers are split across records until merged. Second, the missing
                feed from the legacy asset register leaves the equipment-history panel empty
                on every job screen, which both hides warranty state from technicians and
                makes the panel's audit trail useless for dispute resolution. The invoicing
                mismatch also surfaces here: because completion codes are re-keyed by hand,
                the reporting module and the invoicing system disagree on completed-job counts
                by 2-4% in any given week, and finance reconciles the difference manually at
                month end. The vendor's integration catalogue lists supported connectors for
                both the asset register's database and the invoicing system's import format;
                neither connector has been evaluated. IT's position is that both are
                straightforward but that no integration work was scoped into the rollout
                program, which was staffed for deployment and training only.
                
                6. Program office remarks
                
                Individual regional reports repeat most of the above in local detail. Two
                further observations. First, every negative pattern in this review was visible
                by week four of the Central go-live; the program had no mechanism to convert
                early field observations into program-level fixes, so the same issues
                replayed in each subsequent region. Second, the vendor relationship is good:
                response times are within contract, and both known system gaps have vendor
                solutions waiting on Kestrel decisions rather than on vendor work.
Task

Restructure this review for the COO using the Minto Pyramid Principle. The COO's
decision is whether to extend Fieldbook to North and Islands in Q1.

Эталон судьи gold.md

Gold 10: rollout review

Original gold for an invented scenario.

Expected structure

Top: answers the COO's actual question — extend to North and Islands in Q1,
conditional on fixing what made South and West slow and expensive. A defensible
top must carry both the recommendation and its condition; "the rollout went
well" is a topic, and an unconditional "extend" ignores the document's own
economics (a repeat of the West pattern puts the new regions underwater).

The material honestly supports four first-level branches, not the habitual
three — for example:

  1. The economics already argue for extension — productivity up (4.6 → 5.3
    jobs/day, travel −18%, overtime −12%, fuel −9%), payback reached in the two
    regions that onboarded fast.
  2. Onboarding is the reproducible failure — single classroom slot per
    region left late hires unserved (34 of 118 in West never attended), drove
    44% of support tickets, forced shadowing that cost 380 technician-hours,
    and is the one variable separating payback regions from break-even regions.
  3. Two integration defects tax every region and have waiting fixes — the
    asset-register feed (blind technicians, 9% redundant warranty parts, empty
    audit panel) and the completion-code mismatch (300 records/week re-keyed, a
    temporary clerk, 2–4% reporting discrepancy); vendor connectors and the
    mapping table exist, and stall only because nobody owns integration.
  4. The program lacked a feedback loop — every defect was visible by week
    four of Central and replayed in each region; extension without that loop
    repeats the tax in two more regions.

A grouping that reaches the same content by different cuts is acceptable; what
is not acceptable is a branch per source section (adoption / operations /
support / finance / data), which is the document's order, not an answer.

Section pyramids

The rewrite must give each substantive section its own small pyramid — a
one-sentence section top that its details support — plus the top-level pyramid
whose key line the section tops visibly feed. A single flat pyramid over a
document of this size, with details hanging directly off four branches and the
sections dissolved, loses the layer the reader navigates by. Depth should
follow the material: the integration branch genuinely carries two distinct
defects, each with its own evidence, so it earns a third level; thin branches
must not be padded to match.

Cross-section duplicates (the MECE trap)

The same three facts each appear in two or three sections and must appear
once in the structure, with their scattered evidence gathered:

  • The completion-code mismatch: Operations (300 records/week re-keyed), Support
    (forwarded tickets, vendor's mapping-table answer), Finance (£8,400 clerk),
    Data (2–4% reconciliation gap). One defect, four symptoms.
  • The missing asset-register feed: Operations (blind jobs, 9% redundant parts)
    and Data (empty history panel, useless audit trail). One defect.
  • The training shortfall: Adoption (late hires unserved, shadowing), Support
    (44% of tickets, webinar attendance pattern, the desk lead's comment),
    Finance (380 lost hours). One cause.

An output that keeps any of these as two separate first-level or second-level
points in different branches has failed the cross-section MECE pass.

Facts and proportions

All load-bearing numbers come from the source; invented benchmarks, dates or
vendor quotes are hard failures. The parts-ordering habit, dispatcher uptake,
and vendor-relationship remarks are legitimately minor: demoting them to a
short residual note or explicitly dropping them is proportional omission.
The output is a restructured review, so it will be long — length itself is not
a defect; a section-by-section retelling under new headings is.

Судья gpt-5.6-sol, effort high · Claude Sonnet 5, effort low · навык с коммита f4813ce · прогон v1.7.0-final