Five mechanisms carried the service. I led the design work; where the command center I came from already had a mechanism, we adapted it rather than rebuilt it. The test for each was the same: could someone on shift at 03:00 in another region execute it without calling anyone?
1 · One intake, two doors
Machine door: a service alarm dashboard where a threshold breach opens a high-severity ticket
and pages the on-call automatically, then walks them through the actions for that alarm. Human
door: an escalation matrix, a decision tree that ends in pre-filled ticket links, so an
escalation arrives already classified instead of as a paragraph of prose.
2 · A shared severity language
A prioritisation matrix setting the thresholds and rules for preferential treatment, a list of
vital business functions naming the tools that must keep working, and a glossary. Teams across four organizations
with different vocabularies needed one definition of “bad enough to wake someone”.
3 · Coverage with no seams
A dedicated 24/7 schedule in the corporate on-call tool with daily and weekly redundancy, a
parallel manager-on-duty rota for anything out of the ordinary, and a shift handover with an
explicit acknowledgement handshake and a prioritised handoff board: live high-severity
first, then external events, then everything still being watched.
4 · Notification as a product
An existing three-tier model, extended to the new teams, so “we are telling you something” always carries a known weight, plus the team and leadership distribution lists and the generator that produced the notification itself, both built for the new teams. A committed update cadence per live event, so stakeholders knew when the next update was due instead of asking.
5 · Ownership, written down
A stage-by-stage responsibility split, agreed first with one operations organisation, between its real-time analysts and the command center: intake, validation, severity, monitoring, notification, closure, retrospective. It moved incident admin off the analysts so they could stay on impact analysis while the event was still running, and became the template for the next organisation.
Underneath all five
A catalogue mapping those same 50 systems and tools to the team that owns each one (the taxonomy behind the escalation form), an independent wiki holding the guidelines, and the external-risk tooling the command center already ran, now covering the new teams: Dataminr for physical and outage signals, ArcGIS to work out whether an event actually touches our sites and people.
01 SIGNAL
Alarm threshold breach, or a team escalates through the decision tree
02 PAGE
Ticket opens pre-classified; the on-call is paged automatically
03 ACKNOWLEDGE
First correspondence inside 5 minutes (the measured clock)
04 ROUTE
Validate impact, find the owning service, escalate severity or page the manager on duty
05 ANNOUNCE
Tiered notification out, on a committed update cadence, until closure and retrospective
Early warning
First-touch alert: an event outside the network has confirmed impact, or has raised the risk to operations.
Advisory
Prepare: an event is likely to land, here is what to have ready. Sent to bolster readiness, not to report damage.
Disruption report
The full account of a live disruption (impact, coordination, actions, expected resolution), maintained through the lifecycle of the event.