Case study · shared-service incident response
Manfred Johanning

Amazon Selling Partner Services · 2022–2024

Four organizations,
one incident desk.

Across four seller-trust organizations, most operations teams had no on-call rota, no escalation path, no round-the-clock coverage and no way to announce an outage worldwide. Six incident managers in three regions took them on (40 sites, more than 13,700 people, 50+ critical systems) and moved worldwide service-level attainment from 75% in week 52 of 2022 to 97% by week 5 of 2023.

My role

Manager, Major Incident Response
Command Center, responsive side

Period

Design Mar 2022
Live from week 51, 2022

Team

6 incident managers
3 regions, 24/7

Served

4 seller-trust organizations
40 sites + virtual

2.23min

Average time to acknowledge an event alert on the new service, against a 5-minute goal. Amazon's general SEV2 guideline is 30 minutes.

6min

Average time to resolve or hand off with next steps, against a 30-minute goal.

75 → 97%

Worldwide first-correspondence attainment, week 52 of 2022 to week 5 of 2023.

687

Alerts managed across three regions between the December start and 26 February.

Situation

Incident response existed. It just wasn't shared.

Amazon describes this organisation in its own public job postings as “over 17K Amazonians … spanning 56 global locations, and handling over 50M contacts a year.” It protects the marketplace: account health, payment risk, policy enforcement, brand protection, catalogue integrity. When its tooling breaks, the checks that keep buyers and sellers safe break with it.

One part of that organisation already ran a command center. I had spent the previous decade inside it. The command center's readiness work for Prime Day and the 2021 peak shopping event, which I helped run, is where the gap showed: outside the command center's home organisation, most operations teams had no on-call schedule, no escalation path, no distributed coverage and no worldwide notification mechanism. The teams that took the shared service had mostly been carrying high-severity events on improvisation and goodwill.

Improvisation is expensive in the first ten minutes of an incident, and the first ten minutes are most of the damage.

Constraint

Six people. Four organizations. No authority over any of them.

The teams could not each hire an incident manager, and a central desk could not take uptime accountability away from teams that already owned it. So the service had to be shared, thin and voluntary: six incident managers covering every participating team around the clock, run as a six-month pilot at no cost to the participating teams.

That constraint decided the design. A desk with no authority and no spare capacity cannot be the place where incidents are debated. It has to be the place where an incident is classified, routed and announced: fast, and identically every time. Everything below follows from that one sentence.

It also decided the staffing. The plan began as temporary junior contracts, filled from inside the operations production population. We changed it to permanent program-manager roles, because those contracts could not reliably attract the skill set, judgement and motivation the job needed, and could not be offered on anything but an hourly basis. The reversal went into the plan as a two-way door: if the pilot stopped, the staff would be absorbed back into the parent team. Hiring hubs were placed next to existing command center counterparts in each of the three regions, so new hires could learn by sitting near people who already did the work.

Mechanism

The five mechanisms, and what each one is for.

Five mechanisms carried the service. I led the design work; where the command center I came from already had a mechanism, we adapted it rather than rebuilt it. The test for each was the same: could someone on shift at 03:00 in another region execute it without calling anyone?

1 · One intake, two doors

Machine door: a service alarm dashboard where a threshold breach opens a high-severity ticket and pages the on-call automatically, then walks them through the actions for that alarm. Human door: an escalation matrix, a decision tree that ends in pre-filled ticket links, so an escalation arrives already classified instead of as a paragraph of prose.

2 · A shared severity language

A prioritisation matrix setting the thresholds and rules for preferential treatment, a list of vital business functions naming the tools that must keep working, and a glossary. Teams across four organizations with different vocabularies needed one definition of “bad enough to wake someone”.

3 · Coverage with no seams

A dedicated 24/7 schedule in the corporate on-call tool with daily and weekly redundancy, a parallel manager-on-duty rota for anything out of the ordinary, and a shift handover with an explicit acknowledgement handshake and a prioritised handoff board: live high-severity first, then external events, then everything still being watched.

4 · Notification as a product

An existing three-tier model, extended to the new teams, so “we are telling you something” always carries a known weight, plus the team and leadership distribution lists and the generator that produced the notification itself, both built for the new teams. A committed update cadence per live event, so stakeholders knew when the next update was due instead of asking.

5 · Ownership, written down

A stage-by-stage responsibility split, agreed first with one operations organisation, between its real-time analysts and the command center: intake, validation, severity, monitoring, notification, closure, retrospective. It moved incident admin off the analysts so they could stay on impact analysis while the event was still running, and became the template for the next organisation.

Underneath all five

A catalogue mapping those same 50 systems and tools to the team that owns each one (the taxonomy behind the escalation form), an independent wiki holding the guidelines, and the external-risk tooling the command center already ran, now covering the new teams: Dataminr for physical and outage signals, ArcGIS to work out whether an event actually touches our sites and people.

01  SIGNAL

Alarm threshold breach, or a team escalates through the decision tree

02  PAGE

Ticket opens pre-classified; the on-call is paged automatically

03  ACKNOWLEDGE

First correspondence inside 5 minutes (the measured clock)

04  ROUTE

Validate impact, find the owning service, escalate severity or page the manager on duty

05  ANNOUNCE

Tiered notification out, on a committed update cadence, until closure and retrospective

Early warning

First-touch alert: an event outside the network has confirmed impact, or has raised the risk to operations.

Advisory

Prepare: an event is likely to land, here is what to have ready. Sent to bolster readiness, not to report damage.

Disruption report

The full account of a live disruption (impact, coordination, actions, expected resolution), maintained through the lifecycle of the event.

Rollout

Nothing here launched on a memo.

MAR – APR 2022

Discovery against an eleven-task plan

A working-backwards plan with dated tasks, from standing up the shared-service project to the first day of supervised on-call and a closure meeting. Each candidate team was surveyed for coverage hours, sites, work items, vital tools, business requirements and its own service-level commitments, which ranged from one hour to a week. Work-item verticals spanning roughly 28 sites in the Americas, EMEA and Asia-Pacific.

MAR – NOV 2022

Infrastructure, then hiring

Wiki, escalation matrix, prioritisation matrix, vital business function list, on-call and leadership schedules, distribution lists, notification generator, alarm dashboard. Hiring ran late (candidates needed existing operations knowledge or the ramp was too long), so we opened limited cross-team moves from the existing command center to unblock it.

WEEK 51, 2022

Live across all four organizations

Direct incident management support went live across 40 in-house sites plus virtual sites, over 13,700 employees in EMEA, the Americas, Southeast and East Asia and India, covering 50+ critical systems and external events such as weather and geopolitical disruption.

FEB 2023

Ten roadshows, because adoption is the real launch

Ten sessions with leadership groups from the stakeholder organizations at two sites in Asia, then virtual sessions for the Americas and EMEA, then escalation-process reviews with operations and workforce-management teams. A shared desk only works if every team knows the door exists.

1 MAR 2023

First mock drill

A rehearsed escalation run end to end, recorded and published internally so teams who couldn't attend could still see how an escalation is meant to travel.

MAY 2023

Sign-off package for the next wave

I prepared the status and sign-off document that took the next operations organisation's onboarding into director-level review, with the success measures defined up front (response time, first-touch resolution rate, service-level compliance ratio, active tickets, reopen rate, ratio of root-cause analyses to formal error reports), and formal three- and six-month pilot reviews scheduled before launch rather than after it.

Result

Attainment went up while volume went up four-fold.

Two numbers were measured on every escalation: first correspondence (how long until a human acknowledged it) and time to resolve (how long until it was closed or handed on with next steps). Both were reported weekly as the percentage inside the service level, against a 95% goal and a 90% floor.

Worldwide attainment, week 52 of 2022 to week 6 of 2023 Percentage of escalations inside the service level. Alert count under each week. Axis starts at 70%.
Worldwide service-level attainment by week Two lines, first correspondence and time to resolve, rising from 75 and 74 percent in week 52 of 2022 to 97 and 100 percent in week 5 of 2023. 100 95 90 80 70 GOAL 95% FLOOR 90% 75% 100% 97% W52 53 W01 65 W02 41 W03 69 W04 78 W05 71 W06 76 ALERTS W52: first correspondence 75%, time to resolve 74%, 53 alerts W01: first correspondence 88%, time to resolve 85%, 65 alerts W02: first correspondence 83%, time to resolve 83%, 41 alerts W03: first correspondence 84%, time to resolve 84%, 69 alerts W04: first correspondence 95%, time to resolve 97%, 78 alerts W05: first correspondence 97%, time to resolve 100%, 71 alerts W06: first correspondence 96%, time to resolve 95%, 76 alerts
First correspondence Time to resolve 95% goal / 90% floor
The same series, as reported
WeekFirst corr.Time to resolveAlerts
2022-5275%74%53
2023-0188%85%65
2023-0283%83%41
2023-0384%84%69
2023-0495%97%78
2023-0597%100%71
2023-0696%95%76

The month-over-month view is the one I'd put in front of an operator. December 2022: 75% first correspondence and 73% time to resolve, on 71 alerts. January 2023: 88% and 89%, on 283 alerts. Four times the load, and both numbers up: thirteen points on first correspondence, sixteen on time to resolve. That is the signature of a mechanism working rather than a team trying harder.

Where the improvement actually came from First-correspondence attainment by region, December 2022 against January 2023, with each region's alert count.
First-correspondence attainment by region, December 2022 against January 2023 The third region rises from 36 percent to 68 percent while its alert volume triples; the Americas and Europe hold in the nineties. 0 50 100 Americas December 2022: 100% 100% Americas January 2023: 96% 96% AMERICAS 23 → 98 ALERTS Europe December 2022: 91% 91% Europe January 2023: 95% 95% EUROPE 23 → 111 ALERTS Third region, December 2022: 36% 36% Third region, January 2023: 68% 68% THIRD REGION 25 → 74 ALERTS
December 2022 January 2023

The Americas and Europe were already close to the line. the third region was at 36%, a new team, a new process and the largest share of the growth in volume. It was not a discipline problem; it was an onboarding problem. It reached 68% in January and 100% by week 5 of 2023, and worldwide attainment moved with it. The adoption work that followed through February and March (the roadshows, the escalation-process reviews, the first drill) was aimed at the same gap.

After launch

A service level you don't audit is a rumour.

Across 2023 and 2024, every month, every missed target was reviewed ticket by ticket and given a verdict: a genuine miss, a one-time exception, a mis-attributed resolver, or an allowance for a large-scale event or a still-ramping new hire. Roughly 430 ticket-level decisions across 2023 and 2024. It is unglamorous work, and it is the only reason the percentages above mean anything: the team could not quietly redefine a miss, and a real miss produced a change rather than an argument.

Flagged misses reviewed per month Counts of targets flagged and adjudicated in the monthly audit, August 2023 to August 2024. Counts, not rates. Monthly volume varied.
Flagged service-level misses per month, August 2023 to August 2024 Monthly counts of first-correspondence and time-to-resolve misses reviewed in the audit, falling from 51 and 35 to 9 and 4 with variation in between. 0 25 50 AUG: 51 first correspondence misses AUG: 35 time to resolve misses AUG SEP: 37 first correspondence misses SEP: 21 time to resolve misses SEP OCT: 15 first correspondence misses OCT: 25 time to resolve misses OCT NOV: 4 first correspondence misses NOV: 14 time to resolve misses NOV DEC: 10 first correspondence misses DEC: 17 time to resolve misses DEC JAN: 15 first correspondence misses JAN: 12 time to resolve misses JAN FEB: 15 first correspondence misses FEB: 17 time to resolve misses FEB MAR: 13 first correspondence misses MAR: 18 time to resolve misses MAR APR: 15 first correspondence misses APR: 31 time to resolve misses APR MAY: 27 first correspondence misses MAY: 12 time to resolve misses MAY JUN: 9 first correspondence misses JUN: 7 time to resolve misses JUN JUL: 20 first correspondence misses JUL: 17 time to resolve misses JUL AUG: 9 first correspondence misses AUG: 4 time to resolve misses AUG 2023 2024 51 9
First-correspondence misses Time-to-resolve misses

The same discipline shows up in how the wider organisation ran its peak trading events. For Black Friday and Cyber Monday 2022 (deals running in 20 countries), the control center held a round-the-clock conference bridge with named call leaders from the Friday through the following Wednesday, a daily roll call, hourly red/amber/green status pulled from each operations team at quarter past the hour, one master incident ticket carrying every open issue, and a 30-minute update cadence on anything live, against a fixed status template: issue, customer impact, impacted teams, actions taken, expected resolution, available mitigation levers. Missed check-ins escalated up a named ladder to vice-president level. I was one of six named escalation contacts for the whole control center in the run-up to the event, and the single point of contact for my team in the 41-question readiness assessment (capacity, hiring, operations, readiness, resilience, risk) in both 2022 and 2024.

What this proves

The transferable part, stated plainly.

Shared services, not empires

I can design a central function that serves teams it does not control, and make adoption voluntary by making it obviously easier than the alternative. That is the same problem as a platform team, an internal tools team or a centre of excellence.

Operating mechanisms that outlive me

Intake, severity language, on-call design, notification tiers, a written responsibility split. Each one is a mechanism a new hire can run on their second week, which is the only test that matters for 24/7 coverage.

Measurement with an audit behind it

Two metrics, a goal and a floor, reported weekly by region, with every miss adjudicated monthly. I would rather defend a smaller number that survives scrutiny than publish a larger one that doesn't.

Change management across regions

Four organizations, three regions, ten roadshows, process reviews with operations and workforce management, then a rehearsal before the real thing. Rollout was most of the work; the build was the easy half.

Risk decisions made in the open

Changing the staffing model to de-risk skill fit, with the reversal written down as a two-way door before it was needed. Senior stakeholders sign off on decisions whose downside is already on the page.

Calm at the sharp end

Ten years of high-severity bridges before this programme: outages, cyberattacks, hurricanes. The mechanisms above exist because I know exactly which minute they save.