qwen3.6-fp8 на кейсе «Разбор большого запуска»: с Minto без изменений

Структура 6 → 6 из 8 0 Качество 8 → 8 из 10 0

Судья об ответе с Minto

The concise value-risk-feasibility memo names its order, but compresses away material evidence and flattens integrations.

Режим write · кейс 10-rollout-review · прогон v1.7.0-final

Структура из 8, по 2 балла на ось
СтруктураБез Minto 6С Minto 6Изменение
Вершина22
Состав ключевой линии11
Уровни21−1
Порядок и род12+1
Качество из 10, по 2 балла на ось
КачествоБез Minto 8С Minto 8Изменение
Вершина22
Однородные группы21−1
Объяснимый порядок12+1
MECE11
Видимая соразмерная структура22

Ответы моделей и слова судьи приведены в оригинале, по-английски.

Без Minto

контрольный промпт · слепая метка out-12
  • unacknowledged_source_loss
  • качество со штрафом рубрики 7/10

The document is clear and well nested, but omitting the duplicate-record burden leaves a material part of support and data quality unaccounted for.

RECOMMENDATION

Extend Fieldbook to North and Islands in Q1, conditional on restructuring the onboarding model and executing integration fixes prior to launch.

1. The business case for expansion is validated and economically favorable.

  • Operational efficiency gains are significant and consistent across live regions.
    • Completed jobs per technician rose from 4.6 to 5.3, and average travel time between jobs fell 18%.
    • Same-day emergency insertion is now automatic, reducing median assignment time to 4 minutes.
  • Financial returns confirm payback and positive unit economics.
    • Overtime spending declined 12% quarter-over-quarter, and fuel cost per job dropped 9%.
    • Central and East reached payback within the second quarter; South and West are at break-even.
    • Extension adds £13,500 monthly in licences and requires winter travel for classroom training, but economics remain favourable if adoption velocity matches Central and East.

2. Adoption lags are driven by training logistics, not platform capability, and can be corrected.

  • High adoption correlates directly with comprehensive training coverage.
    • Central and East achieved full adoption within six weeks; all 41 dispatchers reached 100% usage immediately.
    • South and West lagged because the two-day classroom course was scheduled only once per region.
    • In West, 34 of 118 technicians never attended a session, forcing reliance on shadowing that diverted experienced staff from routes.
  • Support volume confirms training gaps rather than product defects.
    • 44% of tickets are password resets concentrated among technicians who missed classroom sessions.
    • The support desk is effectively performing training functions at ticket costs, indicating a need for proactive, continuous onboarding.
  • New regions require a restructured training approach to avoid repeating the West pattern.
    • A single classroom event cannot support rolling hires; a hybrid model with weekly webinars and immediate access is required.
    • Early warnings from Central were not converted into program-level fixes; new regions must implement the revised training model before go-live.

3. Integration gaps create measurable friction but have clear, low-effort resolutions.

  • Missing asset register feed compromises service quality and increases waste.
    • Technicians lack equipment history, leading to unnecessary part replacements in 9% of audited jobs.
    • Back-office calls increase as technicians compensate for missing data.
  • Job-completion code mismatches disrupt billing and reporting accuracy.
    • The billing team manually re-keys approximately 300 records weekly, introducing transcription errors.
    • Reporting and invoicing systems disagree on completed-job counts by 2–4% weekly.
  • Vendor solutions are ready and require only internal execution.
    • Connectors for the asset register and invoicing system exist in the vendor catalogue.
    • IT assesses both integrations as straightforward; the barrier is lack of scope and ownership, not technical complexity.
    • A configurable mapping table for codes can be deployed in roughly one day.
    • Pre-launch actions must assign owners and scope these integrations to protect data integrity and billing efficiency.

С Minto

промпт с навыком · слепая метка out-14
  • invented_facts
  • unacknowledged_source_loss
  • качество со штрафом рубрики 7/10

The concise value-risk-feasibility memo names its order, but compresses away material evidence and flattens integrations.

Subject: Recommendation: Extend Fieldbook to North and Islands in Q1

Fieldbook has delivered strong operational gains and cost reductions across the four live regions, confirming its value to Kestrel's maintenance model. However, rollout defects in training delivery and system integrations are creating avoidable costs, eroding margins in slower regions, and threatening the profitability of any further expansion. Extending to North and Islands in Q1 is recommended, provided we restructure the onboarding program and implement the asset-register and invoicing integrations before go-live.

Fieldbook generates measurable efficiency and cost savings in live regions.

  • Technician productivity increased from 4.6 to 5.3 jobs per day, with average travel time between jobs falling 18%.
  • Emergency same-day insertions are now automatic, reducing median assignment time to 4 minutes.
  • Overtime spending dropped 12% quarter-over-quarter, and fuel cost per job decreased 9%, driven by optimized routing.
  • Dispatchers achieved full adoption by week one following the removal of the legacy spreadsheet.
  • Financial return on investment was realized within the second quarter in Central and East.

Training bottlenecks and integration gaps are driving rework costs and adoption delays.

  • Classroom training is scheduled only once per region, causing adoption lags in South and West; improvised shadowing arrangements consumed an estimated 380 technician-hours of lost route time.
  • Missing equipment service history from the legacy asset register forces technicians to call the back office or ask customers, resulting in 9% of audited jobs replacing parts already covered by warranty.
  • Mismatched job-completion codes between Fieldbook and the invoicing system require the billing team to manually re-key approximately 300 records weekly, introducing transcription errors and a 2–4% monthly reconciliation variance.
  • A search limitation restricted to exact postcodes generates duplicate customer records, requiring support to merge roughly 70 records per week and fragmenting job histories.
  • Program governance failed to escalate negative patterns observed in week four of the Central rollout, allowing identical issues to recur in subsequent regions.

Remediation steps are defined, vendor-supported, and feasible within the Q1 timeline.

  • The vendor offers a configurable mapping table to resolve code mismatches, with setup estimated at one day.
  • Supported connectors exist for both the asset register database and the invoicing import format; IT assessment confirms these integrations are straightforward to implement.
  • Vendor responsiveness remains within contract terms, and all known technical solutions are awaiting Kestrel authorization rather than vendor development.
  • Restructuring onboarding to allow continuous or multiple training sessions will eliminate the shadowing costs and accelerate adoption curves to match Central and East performance.

Next step: Approve Q1 extension contingent on program office scoping the asset-register and invoicing integrations and redesigning the training schedule for North and Islands by end of month.


Restructured into a decision memo with an answer-first top, three ranked groups (value, risk, feasibility), and consolidated cross-section duplicates into MECE branches.

Как устроен тест

qwen3.6-fp8 получила одну и ту же задачу дважды. Контрольный промпт — одна вводная строка и задача. Промпт с Minto — та же задача, перед которой стоит полный текст навыка. Судья читал оба ответа среди 14 ответов под слепыми метками и не знал, в каком из них был навык.

Промпт без Minto 8 317 Б
You are working on a business document using the Minto Pyramid Principle.

<!-- Original fixture. Company, product, regions, people and numbers are invented.
     It tests the large-document behaviors: one pyramid per section plus a
     top-level pyramid, MECE across sections (planted cross-section duplicates),
     and geometry derived from the material rather than a fixed 3x3 habit. -->

# Fixture 10: rollout review

**Mode:** `write`
**Language:** `en`

## Context

Kestrel Maintenance Group services heating and cooling equipment across six
regions. Over the past two quarters it rolled out Fieldbook, a work-order and
scheduling platform, to four of the six regions. The draft below is the internal
review of that rollout, assembled by the program office from status reports. The
reader is the chief operating officer, who must decide at the quarterly meeting
whether to extend Fieldbook to the remaining two regions (North and Islands) in
Q1, and has asked to see this review beforehand.

## Before

```text
FIELDBOOK ROLLOUT — TWO-QUARTER REVIEW (DRAFT)

1. Adoption

Fieldbook is now live in four regions: Central, East, South, and West. Across
these regions 412 of 486 field technicians have activated their accounts, and
83% of work orders are now created and closed inside Fieldbook rather than on
paper or by phone. Central and East reached full adoption within six weeks.
South and West lagged: South took eleven weeks to pass 80% of orders in the
system, and West is still at 71% after nine weeks. Regional coordinators in
both slower regions report the same explanation: the two-day classroom course
was scheduled only once per region, so technicians hired after the course, or
absent during it, learned the platform second-hand from colleagues. In West,
34 of 118 technicians never attended any session. Coordinators improvised
shadowing arrangements, which worked but pulled experienced technicians off
their own routes. Uptake among dispatchers was faster: all 41 dispatchers
across the four regions were using Fieldbook scheduling from week one, since
the old scheduling spreadsheet was switched off on day one and they had no
alternative.

2. Operations

The clearest operational gain is scheduling density. With Fieldbook suggesting
route-aware assignment, completed jobs per technician per day rose from 4.6 to
5.3 across the four live regions, and average travel time between jobs fell
18%. Same-day emergency insertion, which previously required a dispatcher to
phone technicians until someone accepted, is now automatic and takes a median
of 4 minutes. However, the benefit is partly eaten by a data-transfer defect:
Fieldbook does not receive equipment service history from the legacy asset
register, so technicians open each job without knowing what was done at that
site before. They compensate by calling the back office, or by asking the
customer, and in 9% of audited jobs they replaced parts that had already been
replaced within warranty in the past year. The asset register team was asked
in month two for a nightly export; the request is still in their backlog. A
second, smaller friction: job-completion codes in Fieldbook do not match the
codes in the invoicing system, so the billing team re-keys roughly 300 job
records per week into the invoicing system by hand, introducing transcription
errors that customers catch on invoices. Operations leads in Central and East
also note that Fieldbook's parts-ordering screen is unpopular; most
technicians still order parts by phone, though this appears to be habit rather
than a defect, as the screen works correctly in testing.

3. Support and training

The support desk logged 1,940 Fieldbook tickets over the two quarters. Ticket
volume peaked in each region's third week and declined steadily after. 44% of
all tickets are password resets and login problems, concentrated among
technicians who missed the classroom course and had never logged in during the
supported window. Another 19% are duplicate-record tickets: technicians create
a second customer record when they cannot find the first, because the search
screen matches on exact postcode only. The support desk merges these
duplicates manually, at about 70 records per week. Support also fields a
steady trickle of calls from the billing team about mismatched job-completion
codes, which the desk cannot resolve and forwards to the vendor. The vendor's
response on the code mismatch has been that a configurable mapping table
exists and takes roughly a day to set up, but it has not been scheduled
because no one on Kestrel's side owns the integration. Training-wise, the desk
runs a weekly one-hour webinar that reaches 20 to 30 technicians; attendance
correlates strongly with regions that missed classroom slots. The desk lead's
written comment: "We are doing the training program's job at ticket prices —
most of what we handle is people who were never onboarded, plus the two known
system gaps."

4. Finance

Licence and hosting for Fieldbook run £31,000 per month for the four live
regions. Overtime spending in the live regions is down 12% quarter over
quarter, which finance attributes mainly to the scheduling gains, and fuel
cost per job is down 9%, consistent with the travel-time reduction. Against
that, the program has carried unplanned costs: the improvised shadowing
arrangements in South and West consumed an estimated 380 technician-hours of
lost route time, and the billing team added one temporary clerk (£8,400 for
the two quarters) to handle the re-keying of job records into the invoicing
system. The program office estimates that the rollout paid back its running
cost within the second quarter in Central and East, while South and West are
roughly at break-even because of the slower adoption. Extending to North and
Islands would add £13,500 per month in licences and require winter travel for
any classroom training. On the current evidence the economics of extension are
favourable if, and only if, the adoption pace of Central and East can be
reproduced; a repeat of the West pattern would put the new regions underwater
for at least two quarters. The program office therefore leans toward
extending in Q1, provided onboarding is restructured first.

5. Data and integrations

Fieldbook's reporting module is now the source of record for job status across
the four regions, and regional managers use its dashboard in the Monday
review. Data quality inside Fieldbook is generally good, with two exceptions.
First, the customer-record duplicates created through the search limitation
(support merges about 70 per week, as noted) mean that job-history views for
affected customers are split across records until merged. Second, the missing
feed from the legacy asset register leaves the equipment-history panel empty
on every job screen, which both hides warranty state from technicians and
makes the panel's audit trail useless for dispute resolution. The invoicing
mismatch also surfaces here: because completion codes are re-keyed by hand,
the reporting module and the invoicing system disagree on completed-job counts
by 2-4% in any given week, and finance reconciles the difference manually at
month end. The vendor's integration catalogue lists supported connectors for
both the asset register's database and the invoicing system's import format;
neither connector has been evaluated. IT's position is that both are
straightforward but that no integration work was scoped into the rollout
program, which was staffed for deployment and training only.

6. Program office remarks

Individual regional reports repeat most of the above in local detail. Two
further observations. First, every negative pattern in this review was visible
by week four of the Central go-live; the program had no mechanism to convert
early field observations into program-level fixes, so the same issues
replayed in each subsequent region. Second, the vendor relationship is good:
response times are within contract, and both known system gaps have vendor
solutions waiting on Kestrel decisions rather than on vendor work.
```

## Task

Restructure this review for the COO using the Minto Pyramid Principle. The COO's
decision is whether to extend Fieldbook to North and Islands in Q1.

Return only the result, with no explanation of how you produced it.
Промпт с Minto пять частей по порядку · 35 890 Б
  1. Below is a skill written as an instruction. Read it in full and apply it to the task at the end.
  2. ===== SKILL.md ===== SKILL.md на f4813ce 17 235 Б
  3. ===== references/rules.md ===== references/rules.md на f4813ce 6 464 Б
  4. ===== references/templates.md ===== references/templates.md на f4813ce 3 742 Б
  5. ===== TASK ===== вход кейса целиком
  6. Return only the result, with no explanation of how you produced it.
Вход кейса before.md

<!-- Original fixture. Company, product, regions, people and numbers are invented.
It tests the large-document behaviors: one pyramid per section plus a
top-level pyramid, MECE across sections (planted cross-section duplicates),
and geometry derived from the material rather than a fixed 3x3 habit. -->

Fixture 10: rollout review

Mode: write
Language: en

Context

Kestrel Maintenance Group services heating and cooling equipment across six
regions. Over the past two quarters it rolled out Fieldbook, a work-order and
scheduling platform, to four of the six regions. The draft below is the internal
review of that rollout, assembled by the program office from status reports. The
reader is the chief operating officer, who must decide at the quarterly meeting
whether to extend Fieldbook to the remaining two regions (North and Islands) in
Q1, and has asked to see this review beforehand.

Before
FIELDBOOK ROLLOUT — TWO-QUARTER REVIEW (DRAFT)
                
                1. Adoption
                
                Fieldbook is now live in four regions: Central, East, South, and West. Across
                these regions 412 of 486 field technicians have activated their accounts, and
                83% of work orders are now created and closed inside Fieldbook rather than on
                paper or by phone. Central and East reached full adoption within six weeks.
                South and West lagged: South took eleven weeks to pass 80% of orders in the
                system, and West is still at 71% after nine weeks. Regional coordinators in
                both slower regions report the same explanation: the two-day classroom course
                was scheduled only once per region, so technicians hired after the course, or
                absent during it, learned the platform second-hand from colleagues. In West,
                34 of 118 technicians never attended any session. Coordinators improvised
                shadowing arrangements, which worked but pulled experienced technicians off
                their own routes. Uptake among dispatchers was faster: all 41 dispatchers
                across the four regions were using Fieldbook scheduling from week one, since
                the old scheduling spreadsheet was switched off on day one and they had no
                alternative.
                
                2. Operations
                
                The clearest operational gain is scheduling density. With Fieldbook suggesting
                route-aware assignment, completed jobs per technician per day rose from 4.6 to
                5.3 across the four live regions, and average travel time between jobs fell
                18%. Same-day emergency insertion, which previously required a dispatcher to
                phone technicians until someone accepted, is now automatic and takes a median
                of 4 minutes. However, the benefit is partly eaten by a data-transfer defect:
                Fieldbook does not receive equipment service history from the legacy asset
                register, so technicians open each job without knowing what was done at that
                site before. They compensate by calling the back office, or by asking the
                customer, and in 9% of audited jobs they replaced parts that had already been
                replaced within warranty in the past year. The asset register team was asked
                in month two for a nightly export; the request is still in their backlog. A
                second, smaller friction: job-completion codes in Fieldbook do not match the
                codes in the invoicing system, so the billing team re-keys roughly 300 job
                records per week into the invoicing system by hand, introducing transcription
                errors that customers catch on invoices. Operations leads in Central and East
                also note that Fieldbook's parts-ordering screen is unpopular; most
                technicians still order parts by phone, though this appears to be habit rather
                than a defect, as the screen works correctly in testing.
                
                3. Support and training
                
                The support desk logged 1,940 Fieldbook tickets over the two quarters. Ticket
                volume peaked in each region's third week and declined steadily after. 44% of
                all tickets are password resets and login problems, concentrated among
                technicians who missed the classroom course and had never logged in during the
                supported window. Another 19% are duplicate-record tickets: technicians create
                a second customer record when they cannot find the first, because the search
                screen matches on exact postcode only. The support desk merges these
                duplicates manually, at about 70 records per week. Support also fields a
                steady trickle of calls from the billing team about mismatched job-completion
                codes, which the desk cannot resolve and forwards to the vendor. The vendor's
                response on the code mismatch has been that a configurable mapping table
                exists and takes roughly a day to set up, but it has not been scheduled
                because no one on Kestrel's side owns the integration. Training-wise, the desk
                runs a weekly one-hour webinar that reaches 20 to 30 technicians; attendance
                correlates strongly with regions that missed classroom slots. The desk lead's
                written comment: "We are doing the training program's job at ticket prices —
                most of what we handle is people who were never onboarded, plus the two known
                system gaps."
                
                4. Finance
                
                Licence and hosting for Fieldbook run £31,000 per month for the four live
                regions. Overtime spending in the live regions is down 12% quarter over
                quarter, which finance attributes mainly to the scheduling gains, and fuel
                cost per job is down 9%, consistent with the travel-time reduction. Against
                that, the program has carried unplanned costs: the improvised shadowing
                arrangements in South and West consumed an estimated 380 technician-hours of
                lost route time, and the billing team added one temporary clerk (£8,400 for
                the two quarters) to handle the re-keying of job records into the invoicing
                system. The program office estimates that the rollout paid back its running
                cost within the second quarter in Central and East, while South and West are
                roughly at break-even because of the slower adoption. Extending to North and
                Islands would add £13,500 per month in licences and require winter travel for
                any classroom training. On the current evidence the economics of extension are
                favourable if, and only if, the adoption pace of Central and East can be
                reproduced; a repeat of the West pattern would put the new regions underwater
                for at least two quarters. The program office therefore leans toward
                extending in Q1, provided onboarding is restructured first.
                
                5. Data and integrations
                
                Fieldbook's reporting module is now the source of record for job status across
                the four regions, and regional managers use its dashboard in the Monday
                review. Data quality inside Fieldbook is generally good, with two exceptions.
                First, the customer-record duplicates created through the search limitation
                (support merges about 70 per week, as noted) mean that job-history views for
                affected customers are split across records until merged. Second, the missing
                feed from the legacy asset register leaves the equipment-history panel empty
                on every job screen, which both hides warranty state from technicians and
                makes the panel's audit trail useless for dispute resolution. The invoicing
                mismatch also surfaces here: because completion codes are re-keyed by hand,
                the reporting module and the invoicing system disagree on completed-job counts
                by 2-4% in any given week, and finance reconciles the difference manually at
                month end. The vendor's integration catalogue lists supported connectors for
                both the asset register's database and the invoicing system's import format;
                neither connector has been evaluated. IT's position is that both are
                straightforward but that no integration work was scoped into the rollout
                program, which was staffed for deployment and training only.
                
                6. Program office remarks
                
                Individual regional reports repeat most of the above in local detail. Two
                further observations. First, every negative pattern in this review was visible
                by week four of the Central go-live; the program had no mechanism to convert
                early field observations into program-level fixes, so the same issues
                replayed in each subsequent region. Second, the vendor relationship is good:
                response times are within contract, and both known system gaps have vendor
                solutions waiting on Kestrel decisions rather than on vendor work.
Task

Restructure this review for the COO using the Minto Pyramid Principle. The COO's
decision is whether to extend Fieldbook to North and Islands in Q1.

Эталон судьи gold.md

Gold 10: rollout review

Original gold for an invented scenario.

Expected structure

Top: answers the COO's actual question — extend to North and Islands in Q1,
conditional on fixing what made South and West slow and expensive. A defensible
top must carry both the recommendation and its condition; "the rollout went
well" is a topic, and an unconditional "extend" ignores the document's own
economics (a repeat of the West pattern puts the new regions underwater).

The material honestly supports four first-level branches, not the habitual
three — for example:

  1. The economics already argue for extension — productivity up (4.6 → 5.3
    jobs/day, travel −18%, overtime −12%, fuel −9%), payback reached in the two
    regions that onboarded fast.
  2. Onboarding is the reproducible failure — single classroom slot per
    region left late hires unserved (34 of 118 in West never attended), drove
    44% of support tickets, forced shadowing that cost 380 technician-hours,
    and is the one variable separating payback regions from break-even regions.
  3. Two integration defects tax every region and have waiting fixes — the
    asset-register feed (blind technicians, 9% redundant warranty parts, empty
    audit panel) and the completion-code mismatch (300 records/week re-keyed, a
    temporary clerk, 2–4% reporting discrepancy); vendor connectors and the
    mapping table exist, and stall only because nobody owns integration.
  4. The program lacked a feedback loop — every defect was visible by week
    four of Central and replayed in each region; extension without that loop
    repeats the tax in two more regions.

A grouping that reaches the same content by different cuts is acceptable; what
is not acceptable is a branch per source section (adoption / operations /
support / finance / data), which is the document's order, not an answer.

Section pyramids

The rewrite must give each substantive section its own small pyramid — a
one-sentence section top that its details support — plus the top-level pyramid
whose key line the section tops visibly feed. A single flat pyramid over a
document of this size, with details hanging directly off four branches and the
sections dissolved, loses the layer the reader navigates by. Depth should
follow the material: the integration branch genuinely carries two distinct
defects, each with its own evidence, so it earns a third level; thin branches
must not be padded to match.

Cross-section duplicates (the MECE trap)

The same three facts each appear in two or three sections and must appear
once in the structure, with their scattered evidence gathered:

  • The completion-code mismatch: Operations (300 records/week re-keyed), Support
    (forwarded tickets, vendor's mapping-table answer), Finance (£8,400 clerk),
    Data (2–4% reconciliation gap). One defect, four symptoms.
  • The missing asset-register feed: Operations (blind jobs, 9% redundant parts)
    and Data (empty history panel, useless audit trail). One defect.
  • The training shortfall: Adoption (late hires unserved, shadowing), Support
    (44% of tickets, webinar attendance pattern, the desk lead's comment),
    Finance (380 lost hours). One cause.

An output that keeps any of these as two separate first-level or second-level
points in different branches has failed the cross-section MECE pass.

Facts and proportions

All load-bearing numbers come from the source; invented benchmarks, dates or
vendor quotes are hard failures. The parts-ordering habit, dispatcher uptake,
and vendor-relationship remarks are legitimately minor: demoting them to a
short residual note or explicitly dropping them is proportional omission.
The output is a restructured review, so it will be long — length itself is not
a defect; a section-by-section retelling under new headings is.

Судья gpt-5.6-sol, effort high · навык с коммита f4813ce · прогон v1.7.0-final