Define the Problem- If services are completely non-operational, the problem is urgent and requires immediate triage.
- If services are operating but behavior has changed from a known state, the problem is urgent and requires immediate triage.
- If services are operating but artifacts are discovered, the problem requires hand-off for repair.
- In either case, a problem preempts project work. The service-owner must escalate the issue if the problem requires development staff who are on a project timeline.
Communication ChannelsSeverity (Identification)The severity of the issue determines the speed of the response and the process to follow. Severity is a judgment call to be reached between support staff and the service owner. - Severity 1: Service(s) functionality completely disrupted. Use urgent response process.
- Severity 2: Functionality significantly impaired but service(s) available. Use hand-off response process.
- Severity 3: Functionality slightly impaired. Use hand-off response process.
- Severity 4: Cosmetic or edge-case defect. Use hand-off response process.
- Assign staff and respond to Severity 1 issues immediately.
- Assign staff and respond to Severity 2 issues immediately if service is business critical.
- Schedule response to Severity 3 issues accepting impact on project time lines.
- Schedule response to Severity 4 issues avoiding impact on project time lines.
Communication Channels- Identify the service owner. This person is the main point of contact for the problem response or names a designate.
- By whatever channel the problem comes in, the recipient must contact ops-help and the service-owner, if known.in IS&T must contact the service owner and hdnotify@mit.edu.
- For Level I severity:
- The service owner must contact IS&T Senior Staff.
- The service owner then coordinates a conference call with IS&T Senior Staff and the business owner.
- The service owner updates 3Down.
- The service owner can delegate management of triage to specific staff at this point.
- The service owner defines a recipient list: Track the initial
- problem response (technical) thread with issue-tracking mechanism used by the service-owners team; if non-existent ops-help@mit.edu and the ops-help Request Tracker queue
.- Keep the Help Desk in the issue-resolution (customer communications) loop using the
(TBD) - hdnotify@mit.edu email list.
- Identify the service-owner. This person, or designate, is the tech lead for the problem response.
- For non-urgent or longer-term resolutions, the service owner can get a designate from the Help Desk to handle issue resolution, if preferred.
- Add business owners and known announce and support lists
The service owner updates the parties noted here, stakeholders of the affected services, and the initiator of the problem report with an initial communication- The service owner should know who has support responsibilities to the service, across IS&T teams. If an ad-hoc support team must be formed, this must be escalated to senior staff for assignment of permanent support responsibilities (see Close Out, below).
Urgent Response (Resolution)- For a completely unresponsive component, Operations and Infrastructure staff have license to perform basic system operations tasks to get the service back online while attempting to contact the service owner and others (restart, adjust capacity, etc.).
- The service owner or designate updates 3Down in addition to communication channels noted aboveemailing the recipient list.
- When basic system operations are not effective, the service owner (or delegate) takes over as lead of specialist support staff.
- In either case, the service owner validates the service for the problem to be closed.
- Transfer ticket from initial queue (computing-help, ops-help, etc) to the support queue for the service.
- From here, no one should perform any task without direction from the lead.
- Service owner is final arbiter for delegation of tasks, priorities, and timing.
- Service owner (or delegate) and all support staff try to identify the root cause of the problem in addition to restoring service.
- The service and business owners must validate the service for the problem to be closedOnce the service is operating again, attempt to identify the cause of the problem.
- If the resolution is temporary, permanent resolution must be addressed. The service owner decides whether to stay in urgent mode or shift to hand-off mode (below).
- Someone familiar with the system must check to see if restart procedures occurred and if that temporarily resolved the problem.
- If not, Tech Lead or designate restarts component manually, determines if this resolves issue or if a more persistent problem exists.
- Notification: preliminary problem description (and resolution, if applicable) sent to Recipient List:
- initiator of problem ticket, srstaff@mit.edu, the appropriate "announce" list for the service, and if any end-user applications could have been affected, computing-help@mit.edu
- In conjunction with managers currently present, Tech Lead forms Team to troubleshoot issue.
- Emergency Response takes precedence over other project work.
- Tech Lead is project manager for duration of issue resolution.
- Tech Lead is final arbiter for delegation of tasks, priorities, and timing.
- Notification: If resolution is lengthy, Tech Lead will update Recipient List at least once per day of status of resolution.
- Post Mortem: Tech Lead reviews response. If emergency response offers the opportunity for improvement of process, Tech Lead calls a post-mortem with parties who participated in the resolution.
Bug Reports-Handoff- The service owner communicates with the Recipient List and updates 3Down.
- During an extended outage, the service owner or help-desk designate updates the recipient list and 3Down at least daily.
Hand-Off (Bug Reports)- The service owner (or delegate) can move the issue ticket from any initial issue-tracker to the responsible developers' system of record for the service.
- Service owner (or delegate) or help-desk designate notifies the recipient list that the problem has moved into a longer-term resolution, with estimates.
- Service owner
Tech Lead notifies a team leader or manager responsible for each tier of the system affected. - Tech Lead collects information from managers on which mail lists to send notification of issue. This is the Recipient List for this issue. Note it in the ticket.
- Tech Lead and managers determine staff members responsible for issue. This is the Team team.
- Tech Lead sends message to Recipient List notifying them of the issue.
- Team performs preliminary troubleshooting to determine the nature of the issue and identify the staff responsible to remedy the issue.
- Ticket information is transferred or linked to system of record for the issue resolvers.
- Recipient List is notified of the transfer and the managers now responsible for the issue.
Close Out- Managers and business owners to modify team schedules when problem interferes with project schedules.
- This is not a resource request, it is a project-timeline adjustment to compensate for production support.
- Service owner (or delegate) and team communicate relevant technical details and workarounds to the Help Desk (hdnotify@mit.edu), off-band from business owners and end users.
- As resolution moves to different components of the system, hand off tech-lead role to appropriate persons.
- Use Pipeline for additional communications.
Close Out- The service owner documents the following in internal Hermes site for reference: (link TBD)## outages and resolution (including long term action items)## how well the Problem Response process was followed## follow up on resolution long term action items
- Service owner follows up with business owner on long term items
These administrative tasks These things should happen, if they need to, before closing the problem. - If the service owner was unknown or vaguely defined, escalate to senior staff and update Operations ' recordsand Help Desk records (IS&T Service Portfolio TBD).
- If specialist development staff with the right skills were undefined or only temporarily assigned, service owner and senior staff appropriate managers to establish permanent support responsibilities. Operations and Help Desk need a record of these assignments.
- If communications channels to stakeholders (announce or support lists) were ill-defined or incomplete, the service owner must be correct those with sponsors and managers for the service(s).
|