DRAFT Problem-Response Playbook

Define the Severity (Identification)

The severity of the issue determines the speed of the response and the process to follow. Severity is a judgment call to be reached between support staff and the service owner.

• Severity 1: Service(s) functionality completely disrupted. Use urgent response process.
• Severity 2: Functionality significantly impaired but service(s) available. Use hand-off response process.
• Severity 3: Functionality slightly impaired. Use hand-off response process.
• Severity 4: Cosmetic or edge-case defect. Use hand-off response process.

• Assign staff and respond to Severity 1 issues immediately.
• Assign staff and respond to Severity 2 issues immediately if service is business critical.
• Schedule response to Severity 3 issues accepting impact on project time lines.
• Schedule response to Severity 4 issues avoiding impact on project time lines.

Communication Channels

• Identify the service owner. This person is the main point of contact for the problem response or names a designate.
• By whatever channel the problem comes in, the recipient in IS&T must contact the service owner and hdnotify@mit.edu.
• For Level I severity:
  • The service owner must contact IS&T Senior Staff.
  • The service owner then coordinates a conference call with IS&T Senior Staff and the business owner.
  • The service owner updates 3Down.
  • The service owner can delegate management of triage to specific staff at this point.
• The service owner defines a recipient list:
  • Track the problem response (technical) thread with issue-tracking mechanism used by the service-owners team; if non-existent ops-help@mit.edu and the ops-help Request Tracker queue
  • Keep the Help Desk in the issue-resolution (customer communications) loop using the hdnotify@mit.edu email list.
  • For non-urgent or longer-term resolutions, the service owner can get a designate from the Help Desk to handle issue resolution, if preferred.
  • Add business owners and known announce and support lists.
• The service owner should know who has support responsibilities to the service, across IS&T teams. If an ad-hoc support team must be formed, this must be escalated to senior staff for assignment of permanent support responsibilities (see Close Out, below).

Urgent Response (Resolution)

1. For a completely unresponsive component, Operations and Infrastructure staff have license to perform basic system operations tasks to get the service back online while attempting to contact the service owner and others (restart, adjust capacity, etc.).
2. The service owner updates 3Down in addition to emailing the recipient list.
3. When basic system operations are not effective, the service owner (or delegate) takes over as lead of specialist support staff.
   1. Transfer ticket from initial queue (computing-help, ops-help, etc) to the support queue for the service.
   2. From here, no one should perform any task without direction from the lead.
   3. Service owner is final arbiter for delegation of tasks, priorities, and timing.
   4. Service owner (or delegate) and all support staff try to identify the root cause of the problem in addition to restoring service.
4. The service and business owners must validate the service for the problem to be closed.
5. If the resolution is temporary, permanent resolution must be addressed. The service owner decides whether to stay in urgent mode or shift to hand-off mode (below).
6. The service owner communicates with the Recipient List and updates 3Down.
7. During an extended outage, the service owner or help-desk designate updates the recipient list and 3Down at least daily.

Hand-Off (Bug Reports)

1. The service owner (or delegate) can move the issue ticket from any initial issue-tracker to the responsible developers' system of record for the service.
2. Service owner (or delegate) or help-desk designate notifies the recipient list that the problem has moved into a longer-term resolution, with estimates.
   1. Service owner and managers determine staff members responsible for issue. This is the team.
   2. Managers and business owners to modify team schedules when problem interferes with project schedules.
   3. This is not a resource request, it is a project-timeline adjustment to compensate for production support.
3. Service owner (or delegate) and team communicate relevant technical details and workarounds to the Help Desk (hdnotify@mit.edu), off-band from business owners and end users.
4. As resolution moves to different components of the system, hand off tech-lead role to appropriate persons.
5. Use Pipeline for additional communications.

Close Out

1. The service owner documents the following in internal Hermes site for reference: (link TBD)## outages and resolution (including long term action items)## how well the Problem Response process was followed## follow up on resolution long term action items
2. Service owner follows up with business owner on long term items

These administrative tasks should happen, if they need to, before closing the problem.

• If the service owner was unknown or vaguely defined, escalate to senior staff and update Operations and Help Desk records (IS&T Service Portfolio TBD).
• If specialist development staff with the right skills were undefined or only temporarily assigned, service owner and appropriate managers to establish permanent support responsibilities. Operations and Help Desk need a record of these assignments.
• If communications channels to stakeholders (announce or support lists) were ill-defined or incomplete, the service owner must be correct those with sponsors and managers for the service(s).

Problem Response Notes 1242011.JPG

The final version of the IS&T Problem Response Playbook can be found in our knowledge base at: http://kb.mit.edu/confluence/x/ioK2