This guide draws on the Vitaver Playbook’s Mission First principle. Name the outcome the business must protect, then decide what to keep running during the interruption.
Choose the service the business must preserve
When a system goes down, the first decision is which customer or operational promise still needs to be kept. An airline may need to tell stranded passengers what to do next. A distributor may need to receive urgent orders and keep a reliable record of what it promises to ship. A finance team may need to identify pending payments while protecting customers from duplicate charges. The service owner should define a minimum service that staff can deliver safely while engineers work on recovery.
The risk is concrete. In its review of the July 19, 2024 outage, CrowdStrike identified a Windows sensor content update that caused a widespread disruption. Delta’s July 24 customer update described its continued recovery. The episode prompts a useful question for any service owner. If a shared component fails across the business, which commitment can staff still make and keep?
Prevent the issues you can contain
Start with changes under your control. Ask the application or endpoint owner to identify updates that could disrupt many users at once, including application releases, security configuration, and vendor updates. For changes you can stage, start with a small group that reflects the real operating environment. Define beforehand what stops the rollout, who can halt it, and what rollback or recovery requires. Verify an actual customer task and include a check for failure from a different device or location. A healthy server alone tells you little about whether a customer can submit an order.
Microsoft’s safe deployment guidance recommends progressive exposure with health checks between stages. Ask third-party providers what control you have over rollout timing, how you learn about a bad update, and how you would recover a device that cannot start. Agree on a response for urgent changes so that delaying a security update has an owner and an explicit review time.
Define the limits of the fallback
NIST’s contingency planning overview includes alternate equipment, alternate sites, and manual processing for short interruptions. Choose the fallback based on the work. For a regional distributor, a limited intake queue can keep urgent requests visible while order processing is unavailable. Give staff an approved form with a unique reference, arrival time, priority, owner, and a way to reach the requester. Keep it in an approved alternate system or a controlled paper register with a named custodian.
Publish three lists in the action card. Which requests can staff accept? Which commitments require an independent check of inventory, identity, price, or payment status? Which requests must wait? If availability cannot be verified, staff can acknowledge receipt and give a time for an update. The logistics owner can then prioritize shipments using records the team can verify. Name a trigger for reducing the scope further if the backlog becomes larger than the staff can safely handle.
Return to normal with a reconciled queue
Decide before the outage how the temporary queue will go back into the main system. Use a unique reference for each case, record who entered or approved it, and distinguish accepted requests from completed work. When the system returns, pause new writes to the fallback queue at an agreed time, preserve the ledger, and check each item against the restored system before entering it. Hold ambiguous items for a named reviewer. For a payment in an uncertain state, have an authorized person verify its status using the approved source before anyone retries it.
Write a one-page plan for the first operational shift. Include the activation trigger, who can activate it, the approved tools, what employees can promise, and when the next update goes out. In the next rehearsal, ask the deputy service owner to activate the fallback, count the open cases, and reconcile a sample with the restored system. The process is ready when staff can account for every sampled request and explain who owns the remainder.