gpt-oss-120b на кейсе «Выжимка из нескольких источников»: с Minto без изменений

Структура 4 → 4 из 8 0 Качество 5 → 5 из 10 0

Судья об ответе с Minto

The top reverses the register’s pricing status, misstates ownership of the diagram, and loses the Lisbon exposure.

Режим digest · кейс 09-sources-digest · прогон v1.7.0-final

Структура из 8, по 2 балла на ось
СтруктураБез Minto 4С Minto 4Изменение
Вершина10−1
Состав ключевой линии11
Уровни11
Порядок и род12+1
Качество из 10, по 2 балла на ось
КачествоБез Minto 5С Minto 5Изменение
Вершина10−1
Однородные группы12+1
Объяснимый порядок12+1
MECE10−1
Видимая соразмерная структура11

Ответы моделей и слова судьи приведены в оригинале, по-английски.

Без Minto

контрольный промпт · слепая метка out-02
  • invented_facts
  • качество со штрафом рубрики 4/10

Exhaustive coverage becomes a source-retelling document with duplicated rows and actions rather than a one-screen synthesis.

Bottom line:
Aldergate has answered most policy‑level questions, but the concrete artefacts needed to satisfy compliance (approved‑package list, architecture diagram, and the resulting storage/encryption/logging details) are still missing; without them Fernway cannot finalize the build‑phase price or confirm that all security requirements are met.


What is already answered

QuestionAnswer (from Aldergate)
Q1 – Is a security review required before production?Yes. A mandatory review will decide if a penetration test is also needed.
Q2 – Is there a security standard or checklist to build against?An internal hardening framework exists, but a review is still required regardless of conformance.
Q3 – Policy on open‑source dependencies?A policy exists; an approved‑package list will be supplied (owner = D. Mercer, dated 12 May).
Q4 – Must critical dependencies be approved individually?Dependency usage is assessed during the security review (no per‑dependency pre‑approval required).
Q5 – Turnaround time for dependency approval?No standard turnaround – approvals are handled case‑by‑case.
Q6 – Can Fernway staff work from outside the UK?Work from Ireland is acceptable (no other countries mentioned).
Q7 – Where may customer data be stored, with what encryption & logging?Not answerable yet – requires an architecture diagram and design walkthrough (encryption/logging requirements exist but are context‑dependent).

Additional context from the July sync:

  • The approved‑package list is “being prepared” and will be sent “shortly” (action assigned to R. Whitcombe, no date).
  • S. Okafor is now the information‑security owner (has not yet attended a joint meeting).
  • Fernway will send a draft architecture diagram before the next sync to enable a concrete answer to Q7.
  • Two Fernway engineers are based in Lisbon; Aldergate gave no response regarding that location.

What still has to be obtained from Aldergate

ItemWhy it’s neededCurrent status
Approved‑package listTo verify open‑source dependencies against the bank’s policy and to close R‑11.Promised 12 May (Mercer) → promised again 9 July (Whitcombe); still not received (98 days open).
Final architecture diagram & design walkthroughRequired to answer Q7 (data‑storage location, encryption, logging) and to complete the security review.Draft exists (since 11 Aug) but has not been sent to Aldergate (R‑14).
Concrete storage, encryption & logging specificationsNeeded for compliance evidence and to determine any additional controls.Dependent on the architecture diagram; pending.
Confirmation on staff locations beyond Ireland (e.g., Lisbon engineers)To ensure remote work complies with any location‑specific rules.No response from Aldergate; open.
Schedule for the mandatory security review (and possible penetration test)To lock in the build‑phase timeline and pricing.Not yet defined; open.

Immediate actions for today

  1. Escalate the approved‑package list request to R. Whitcombe (or directly to D. Mercer) with a firm deadline (e.g., end of week).
  2. Send the draft architecture diagram to Aldergate (copy S. Okafor) and request a design walkthrough slot before the next sync.
  3. Ask Aldergate to confirm whether Lisbon‑based engineers are acceptable or if additional approvals are needed.
  4. Request a tentative schedule for the mandatory security review (and any subsequent penetration test).

Completing these items will unblock R‑07, R‑11, and R‑14, allowing Fernway to finalize the build‑phase price and proceed with compliance‑aligned development.

С Minto

промпт с навыком · слепая метка out-14
  • invented_facts

The top reverses the register’s pricing status, misstates ownership of the diagram, and loses the Lisbon exposure.

Answer: We can price the build phase now that the mandatory security review and most policy questions are clear, but we must still obtain the approved‑package list and the architecture diagram (to resolve data‑storage and encryption requirements) before the compliance track can be closed.

What’s already answered (Situation + Complication)

  • Security review is required and will decide if a penetration test is needed (Source 1 Q1).
  • A hardening framework exists, but a review is still required regardless of conformance (Source 1 Q2).
  • Open‑source policy and dependency handling are defined: a policy exists, critical dependencies are assessed during the review, and staff may work from Ireland (Source 1 Q3‑Q6).
  • Turnaround for dependency approval is case‑by‑case, not fixed (Source 1 Q5).

What still needs to be obtained (what the tech lead must act on)

  • Approved‑package list – promised on 12 May (Mercer) and again on 9 July (Whitcombe) but not yet received (Source 2; Source 3 R‑11).
  • Architecture diagram – required to answer where customer data can be stored and the encryption/logging requirements; draft exists since 11 Aug but has not been sent to Aldergate (Source 2; Source 3 R‑14).
  • Storage‑and‑encryption details – cannot be answered without the architecture diagram and a design walkthrough (Source 1 Q7).

Immediate actions for today

  1. Follow up with S. Okafor (information‑security owner) to expedite the approved‑package list.
  2. Send the draft architecture diagram to Aldergate before the next sync to unlock the storage‑encryption answer.
  3. Record the pending items (approved‑package list, architecture diagram) in the risk register as blockers for pricing.

Как устроен тест

gpt-oss-120b получила одну и ту же задачу дважды. Контрольный промпт — одна вводная строка и задача. Промпт с Minto — та же задача, перед которой стоит полный текст навыка. Судья читал оба ответа среди 16 ответов под слепыми метками и не знал, в каком из них был навык.

Промпт без Minto 3 667 Б
You are working on a business document using the Minto Pyramid Principle.

<!-- Original fixture. The scenario, company, people, systems and numbers are
     invented for this benchmark. The defect pattern it tests — a digest of several
     read sources delivered answer-first into chat — is modeled on real usage. -->

# Fixture 09: sources digest

**Mode:** `digest`
**Language:** `en`

## Context

You are assisting the tech lead of Fernway Systems, a vendor building a customer
support assistant for Aldergate Bank. Before Fernway can price the build phase, the
bank's compliance office must confirm its security requirements. You have just read
the three sources below in full. The tech lead has not read them and will not; they
are deciding what to do about the compliance track today.

## Before

Source 1 — `gate-review.xlsx`, sheet "Security questions", filled in by Aldergate's
compliance office on 12 May:

```text
Q1 Is a security review required before production?
   A: Yes. A review is mandatory; whether a penetration test is also needed is
   decided during the review itself, by the assigned reviewer.
Q2 Is there a security standard or checklist we can build against?
   A: An internal hardening framework exists; a review is still required
   regardless of conformance.
Q3 What is the policy on open-source dependencies?
   A: A policy exists. The approved-package list will be sent to Fernway.
   Owner: D. Mercer. Date: 12 May.
Q4 Must critical dependencies be approved individually?
   A: Dependency usage is assessed during the security review.
Q5 What is the turnaround time for dependency approval?
   A: There is no standard turnaround; it is case by case.
Q6 Can Fernway staff work from outside the UK?
   A: Work from Ireland is acceptable. (No other country is mentioned.)
Q7 Where may customer data be stored, and with what encryption and logging?
   A: Declined — "cannot be answered without an architecture diagram and a
   design walkthrough". Encryption and logging requirements exist but are
   not applicable outside an architectural context.
```

Source 2 — follow-up note from the 9 July delivery sync, `sync-0709.md`:

```text
Attendees: R. Whitcombe (Aldergate, delivery), tech lead (Fernway), PM (Fernway).
- Whitcombe repeated that the approved-package list "is being prepared" and will
  reach Fernway "shortly". Action on Whitcombe, no date attached.
- Compliance contact named for the first time: S. Okafor, information security
  owner. Okafor has not attended any joint meeting so far; Whitcombe passes
  questions along.
- Agreed: Fernway sends the draft architecture diagram before the next sync so
  the storage and encryption questions can get a real answer.
- Fernway's PM noted two Fernway engineers are based in Lisbon. Nobody from
  Aldergate responded to the remark in the meeting.
```

Source 3 — Fernway's internal risk register, rows touching compliance, as of
18 August:

```text
R-07  Security requirements unconfirmed; build-phase price cannot be fixed.
      Owner: tech lead. Status: open since 3 May.
R-11  Approved-package list not received. First promised 12 May (Mercer, per
      gate review), promised again 9 July (Whitcombe). Status: open, 98 days.
R-14  Architecture diagram not yet sent to Aldergate; blocks the data-storage
      answer (gate review Q7). Draft exists since 11 August. Status: open.
```

## Task

Give the tech lead a digest of these three sources in chat: what is already
answered on the compliance track, and what still has to be obtained from
Aldergate. The tech lead will act on it today.

Return only the result, with no explanation of how you produced it.
Промпт с Minto пять частей по порядку · 31 240 Б
  1. Below is a skill written as an instruction. Read it in full and apply it to the task at the end.
  2. ===== SKILL.md ===== SKILL.md на f4813ce 17 235 Б
  3. ===== references/rules.md ===== references/rules.md на f4813ce 6 464 Б
  4. ===== references/templates.md ===== references/templates.md на f4813ce 3 742 Б
  5. ===== TASK ===== вход кейса целиком
  6. Return only the result, with no explanation of how you produced it.
Вход кейса before.md

<!-- Original fixture. The scenario, company, people, systems and numbers are
invented for this benchmark. The defect pattern it tests — a digest of several
read sources delivered answer-first into chat — is modeled on real usage. -->

Fixture 09: sources digest

Mode: digest
Language: en

Context

You are assisting the tech lead of Fernway Systems, a vendor building a customer
support assistant for Aldergate Bank. Before Fernway can price the build phase, the
bank's compliance office must confirm its security requirements. You have just read
the three sources below in full. The tech lead has not read them and will not; they
are deciding what to do about the compliance track today.

Before

Source 1 — gate-review.xlsx, sheet "Security questions", filled in by Aldergate's
compliance office on 12 May:

Q1 Is a security review required before production?
                   A: Yes. A review is mandatory; whether a penetration test is also needed is
                   decided during the review itself, by the assigned reviewer.
                Q2 Is there a security standard or checklist we can build against?
                   A: An internal hardening framework exists; a review is still required
                   regardless of conformance.
                Q3 What is the policy on open-source dependencies?
                   A: A policy exists. The approved-package list will be sent to Fernway.
                   Owner: D. Mercer. Date: 12 May.
                Q4 Must critical dependencies be approved individually?
                   A: Dependency usage is assessed during the security review.
                Q5 What is the turnaround time for dependency approval?
                   A: There is no standard turnaround; it is case by case.
                Q6 Can Fernway staff work from outside the UK?
                   A: Work from Ireland is acceptable. (No other country is mentioned.)
                Q7 Where may customer data be stored, and with what encryption and logging?
                   A: Declined — "cannot be answered without an architecture diagram and a
                   design walkthrough". Encryption and logging requirements exist but are
                   not applicable outside an architectural context.

Source 2 — follow-up note from the 9 July delivery sync, sync-0709.md:

Attendees: R. Whitcombe (Aldergate, delivery), tech lead (Fernway), PM (Fernway).
                - Whitcombe repeated that the approved-package list "is being prepared" and will
                  reach Fernway "shortly". Action on Whitcombe, no date attached.
                - Compliance contact named for the first time: S. Okafor, information security
                  owner. Okafor has not attended any joint meeting so far; Whitcombe passes
                  questions along.
                - Agreed: Fernway sends the draft architecture diagram before the next sync so
                  the storage and encryption questions can get a real answer.
                - Fernway's PM noted two Fernway engineers are based in Lisbon. Nobody from
                  Aldergate responded to the remark in the meeting.

Source 3 — Fernway's internal risk register, rows touching compliance, as of
18 August:

R-07  Security requirements unconfirmed; build-phase price cannot be fixed.
                      Owner: tech lead. Status: open since 3 May.
                R-11  Approved-package list not received. First promised 12 May (Mercer, per
                      gate review), promised again 9 July (Whitcombe). Status: open, 98 days.
                R-14  Architecture diagram not yet sent to Aldergate; blocks the data-storage
                      answer (gate review Q7). Draft exists since 11 August. Status: open.
Task

Give the tech lead a digest of these three sources in chat: what is already
answered on the compliance track, and what still has to be obtained from
Aldergate. The tech lead will act on it today.

Эталон судьи gold.md

Gold 09: sources digest

Original gold for an invented scenario.

Expected structure

Top: an answer to the tech lead's question, not a description of the sources.
The strongest available top is an action with its ground: the blocking questions
cannot be answered by more reading — send the architecture diagram (drafted and
sitting unsent since 11 August) and escalate the approved-package list as an
obligation now 98 days old. A top that merely states "five of seven questions are
answered" is weaker but acceptable if the two actions lead the groups.

Three same-kind groups the material supports, in any defensible order:

  1. Settled — do not re-ask. Review is mandatory (pen test decided during it);
    a hardening framework exists but exempts nothing; dependencies are assessed in
    the review with no standard turnaround; Ireland is cleared for remote work.
  2. Blocked on Fernway's own move. The storage/encryption/logging answer (Q7)
    was declined pending an architecture diagram; the draft has existed since
    11 August and has not been sent (R-14). This is Fernway's blocker, not
    Aldergate's.
  3. Owed by Aldergate and overdue. The approved-package list, first promised
    12 May by Mercer per the gate review — not 9 July by Whitcombe, so 98 days
    overdue, which makes the ask an escalation of an old commitment rather than a
    reminder. Direct access to Okafor, who has never attended and is reachable
    only through Whitcombe.
Cross-source synthesis the digest must make
  • The package-list promise appears in two sources with two dates; the digest
    must date it from 12 May (the older commitment), not 9 July.
  • Q7's "declined" plus R-14's "draft exists since 11 August" must combine into
    "the ball is in Fernway's court"; either fact alone misses the point.
  • The Lisbon remark: Q6 clears Ireland only. Two engineers in Lisbon are an
    unanswered exposure, not a settled fact. Presenting Lisbon as approved is an
    invention; omitting it entirely loses the one new risk the sources surface.
Presentation

Chat prose for one reader: answer first, grouped support, a reference to which
source each load-bearing claim comes from (sheet, note, or register row —
exact citation format is free). No marker legend, no mermaid diagram, no
findings table, no meta-commentary about how the groups are ordered. Headings or
bold group labels are acceptable; a full document apparatus (SCQ labels as
headings, scoring, appendix) is excessive. Length: roughly one chat screen —
condensing the six settled answers into one or two lines each is proportional
omission, not source loss.

Судья gpt-5.6-sol, effort high · навык с коммита f4813ce · прогон v1.7.0-final