Page History

Define the Problem

• If services are completely non-operational, the problem is urgent and requires immediate triage.
• If services are operating but behavior has changed from a known state, the problem is urgent and requires immediate triage.
• If services are operating but artifacts are discovered, the problem requires hand-off for repair.
• In either case, a problem preempts project work. The service-owner must escalate the issue if the problem requires development staff who are on a project timeline.

Communication Channels

• Identify the service-owner. This person is the Tech Lead for the problem response or names a designate.
• By whatever channel the problem comes in, the recipient must contact ops-help and the service-owner, if known.
• The Tech Lead defines a Recipient List
  • Track the initial problem response (technical) thread with the ops-help@mit.edu and the ops-help Request Tracker queue.
  • Keep the Help Desk in the issue-resolution (customer communications) loop using the (TBD) email list.
• Identify the service-owner. This person, or designate, is the tech lead for the problem response.
• The service owner updates the parties noted here, stakeholders of the affected services, and the initiator of the problem report with an initial communication.
• • Add business owners and known announce and support lists.
• The tech lead The service owner should know who has support responsibilities to the service. If an ad-hoc support team must be formed, this must be escalated to senior staff for assignment of permanent support responsibilities (see Close Out, below).

Urgent Response (Resolution)

1. For a completely unresponsive component, Operations and Infrastructure staff have license to perform basic system operations tasks to get the service back online while attempting to contact the service owner and others (restart, adjust capacity, etc.).
2. The service owner or designate tech lead updates 3Down in addition to communication channels noted abovethe Recipient List.
3. When basic system operations are not effective, the service owner tech lead takes over as lead of specialist support staff.
   1. In either case, the service owner validates the service for the problem to be closed.
   2. Once the service is operating again, attempt to identify the cause of the problem.
   3. 1. From here, no one should perform any task without direction from the lead.
      2. Tech Lead is final arbiter for delegation of tasks, priorities, and timing.
      3. The tech lead and all support staff try to identify the root cause of the problem in addition to restoring service.
   4. The service and business owners validates the service for the problem to be closed.
   5. If the resolution is temporary, permanent resolution must be addressed. The service owner decides whether to stay in urgent mode or shift to hand-off mode (below).
   1. 1. Someone familiar with the system must check to see if restart procedures occurred and if that temporarily resolved the problem.
      2. If not, Tech Lead or designate restarts component manually, determines if this resolves issue or if a more persistent problem exists.
   2. Notification: preliminary problem description (and resolution, if applicable) sent to Recipient List:
      1. initiator of problem ticket, srstaff@mit.edu, the appropriate "announce" list for the service, and if any end-user applications could have been affected, computing-help@mit.edu
   3. In conjunction with managers currently present, Tech Lead forms Team to troubleshoot issue.
      1. Emergency Response takes precedence over other project work.
      2. Tech Lead is project manager for duration of issue resolution.
      3. Tech Lead is final arbiter for delegation of tasks, priorities, and timing.
   4. Notification: If resolution is lengthy, Tech Lead will update Recipient List at least once per day of status of resolution.
   5. Post Mortem: Tech Lead reviews response. If emergency response offers the opportunity for improvement of process, Tech Lead calls a post-mortem with parties who participated in the resolution.

   Bug Reports-Handoff

   1. The service owner communicates with the Recipient List and updates 3Down.
   2. During an extended outage, the service owner updates the Recipient List and 3Down at least daily.

   Bug Reports-Handoff

   1. The Tech Lead can move the issue ticket from the ops-help queue to the developers' system of record for the service.
   2. Tech Lead notifies Pipeline representative for his area so resource needs can be taken to Pipeline.
   3. Tech Lead notifies the Recipient List that the problem has moved into a longer-term resolution, with estimatesTech Lead notifies a team leader or manager responsible for each tier of the system affected. Tech Lead collects information from managers on which mail lists to send notification of issue. This is the Recipient List for this issue. Note it in the ticket.
      1. Tech Lead and managers determine staff members responsible for issue. This is the Team.
   4. Tech Lead sends message to Recipient List notifying them of the issue.
   5. Team performs preliminary troubleshooting to determine the nature of the issue and identify the staff responsible to remedy the issue.
   6. Ticket information is transferred or linked to system of record for the issue resolvers.
   7. 1. Pipeline representatives work with managers and business owners to modify Team schedules when problem interferes with project schedules.
      2. This is not a resource request, it is a project-timeline adjustment to compensate for production support.
   8. Tech Lead and Team communicate relevant technical details and workarounds to the Help Desk, off band from business owners and end users.
   9. As resolution moves to different components of the system, hand off tech-lead role to appropriate persons.
   10. Use Pipeline for resolution progress and to manage additional communicationsRecipient List is notified of the transfer and the managers now responsible for the issue.

   Close Out

   These things should happen, if they need to, before closing the problem.

   • If the service owner was unknown or vaguely defined, escalate to senior staff and update Operations' records.
   • If specialist development staff with the right skills were undefined or only temporarily assigned, service owner and senior staff appropriate managers to establish permanent support responsibilities. Operations needs a record of these assignments.
   • If communications channels to stakeholders (announce or support lists) were ill-defined or incomplete, the service owner must be correct those with sponsors and managers for the service(s).