Incident Management Procedure
1. Purpose
The purpose of this procedure is to define the manner in which Incidents are recorded, classified, prioritised, escalated (if needed), resolved and closed in compliance with the requirements of ISO 20000.
An Incident is defined as ‘an unplanned interruption to an IT service or a reduction in the quality of an IT Service’.
2. Scope
This policy applies to all Digital Space employees, contractors and personnel for or on behalf of Digital Space.
The AWS Support page provides further information on procedures for managing incidents from customers using AWS services.
3. Roles and Responsibilities
The following key roles are required to support the Incident Management Procedure:
3.1. Senior Management
Responsibilities include:
- Monitoring the effectiveness of incident management;
- Providing effective communications to business stakeholders, including management information and reports;
- Developing and maintaining the incident management processes.
3.2. Technical Support Team Leads and Managers
These personnel are responsible for coordinating the activities of the triage teams in the effective resolution of an incident. They ensure that the Incident Management procedure is followed, and any supporting actions are documented.
Responsibilities include:
- Driving the efficiency and effectiveness of the incident management process;
- Providing effective communications to business stakeholders, including management information
- and reports;
- Effective escalation to Senior Management;
- For P0s and P1s the Technical Support Team will provide root cause analysis reports to the customers.
3.3. Triage
The following teams can be the first point of contact for our users needing when there is a service
disruption :
- Technical Support Teams.
The Triage engineer initially decides whether it is a Service Request or an Incident. If it is defined as an incident, the triage engineer has the following responsibilities:
- Collect as much usable information as possible;
- Classify, and prioritise the incident;
- Confirm that the correct contact is recorded for the incident;
- Create a short description of the incident including the affected Product;
- Assign to the appropriate team / engineer.
3.4. First Line Support
The First Line support teams have the following responsibilities:
- Provide initial investigation into any emerging incident, validating the customer information;
- Review the assigned priority and update if necessary;
- Act as the source of incident communications to affected stakeholders, supplying guidance and advice issued from the Support Teams;
- Investigate a solution and plan actions required to apply a fix where relevant;
- Conduct or assist in effective testing of an applied fix or resolution as part of incident resolution where required;
- Engage with Field Engineering as required who would manage their action as per the Field Engineering Work Order process;
- If unable to resolve, escalate to second / third line support within the associated technology tower;
- Changes required to support timely service restoration would be managed as Emergency changes and reviewed by the CAB retrospectively.
- Document actions taken within the incident record.
- Provide customer updates in line with the Digital Space Service Level Agreement Policy, via the Incident record and telephone where appropriate.
3.5. Second Line Support
They have the following responsibilities:
- Liaise with the First Line Team to gain an understanding of the background information relating to the service request. The incident record is also utilised to gain additional knowledge;
- Understand the urgency of the incident;
- Act as the source of incident communications to affected stakeholders, supplying guidance and advice issued from the Support Teams;
- Investigate a solution and plan actions required to apply a fix where relevant;
-
Conduct or assist in effective testing of an applied fix or resolution as part of incident resolution where required;
-
Engage with Field Engineering as required who would manage their action as per the Field Engineering Work Order process;
-
Changes required to support timely service restoration would be managed as Emergency changes and reviewed by the CAB retrospectively.
-
Document all actions taken within the incident record.
-
Provide customer updates in line with the Digital Space Service Level Agreement Policy, via the Incident record and telephone where appropriate.
3.6. Third line Support
Third line teams have the following responsibilities:
- Validate, investigate and progress the diagnosis of an Incident when escalated from First- or Second-Line Teams;
- Actively assist in the triage and restoration of services;
- Offer advice for work around strategies and tactical fixes;
- Document actions taken within the Incident record;
- Validate any applied fixes or work rounds;
- Contribute to the effective communication of circumstantial changes during an incident;
- Act as a point of technical escalation for another technical team;
- Liaise with third parties and service support teams to restore service;
- Engage with Field Engineering as required who would manage their action as per the Field Engineering Work Order process;
- Changes required to support timely service restoration would be managed as Emergency changes and reviewed by the CAB retrospectively.
- Provide customer updates in line with the Digital Space Service Level Agreement Policy, via the Incident record and telephone where appropriate.
4. Procedure
4.1. Receiving and Raising Incidents
4.1.1. Hours of service
The Technical Support Teams provide a service from 8am to 6pm unless the customer contract states
that a 24x7x365 service is provided.
4.1.2. Communication Channels
The following channels of communication are employed to raise an incident:
For customer support incidents:
• Log tickets in https://portal.digitalspace.co.uk;
• Email to [email protected] (Network Operations Centre);
• Phone 0333 220 0222
Tickets are also raised automatically as a result of proactive input from monitoring.
4.1.3. Triage
There are 2 levels of Triage.
- Triage engineer decides whether it is a Service Request or an Incident.
- Once defined as an incident, the Triage engineer will:
- Identify what service(s), customers(s), device(s) or application(s) are affected;
- Confirm that it is a valid issue;
- Confirm that we have the correct contact for the incident;
- Identify who/ how many users are affected;
- Identify the impact from service(s), device(s) or application(s) affected.
- Classify, and prioritise the incident;
- Create a short description of the incident including the affected Product;
- Assign to the appropriate team.
The incident is categorised based upon the information received using a P1 to P4 matrix as outlined in the Digital Space Service Level Agreement Policy (this categorises the issue by the service or system area that is affected) taking into consideration Impact and Urgency.
The incident is saved as a ticket within the IT Service Management System which triggers an automatic email to be sent to the caller, notifying them that the Incident has been logged and providing them with a reference number.
Response times are defined relative to the priority of the incident. Standard Product-SLAs are defined within the IT Service Management System.
The priority is automatically defined for incidents raised from proactive monitoring.
4.2. Progression of Incident
The following actions are conducted:
- The priority is reviewed.
- If the incident, once reviewed, is not deemed to be a P1, the first line engineer initially investigates the incident.
- If deemed a P1 incident, this is routed to the Technical Support teams to invoke the Major Incident Management Procedure.
- If the first line engineer is unable to resolve, the incident is passed to a second / third line support within the associated technology tower.
- Customers are provided with updates at a frequency relative to the SLA priority within ServiceNow. Incidents can be put on hold or escalated if required.
- The ticket owner is responsible for ensuring that any subsequent detailed investigation and diagnosis is logged within the ticket. Once the incident is deemed to be resolved, this must be verified with the customer. Once resolution has been verified, the ticket is resolved and will automatically close after 5 days providing no further correspondence with the customer.
5. Service Reviews and Reporting
The following statistics are available to be reported against broken down, where required by day, week, month, year to date and/or year:
- Number of calls;
- Number of incidents (open and closed);
- Number of service requests (open and closed);
- Associated SLAs.
On a Monthly basis, an internal Service Report is produced by the Customer Experience Director and Technical Services Director for the Operations Board to review the previous month’s logged incidents focusing on the following detail:
- A review of all incidents raised including the volume of incidents, priorities and resolutions against the defined SLAs;
- Commentary regarding any missed SLA (e.g., reasons and details of corrective action taken and any suggested improvements etc.);
- Commentary regarding any peaks, dips, trends, reasons, details of corrective action taken and any suggested improvements etc.;
- Customer Satisfaction response rates and results;
- Improvement suggestions from customers.
Service Review reports are created on a per customer basis by the customer Service Manager. These are tailored for each customer.
The service reviews, reporting and analysis enables the business to review the supply of services versus the demand for services and appropriately assign resource (human, technical, financial and/or information) in order to provide the service levels required to meet the needs and expectations of users.
6. Incident Management Analysis
Root Couse Analysis Reports are conducted for all P1 incidents by the Major Incident Manager, to identify the root cause of the incident, the resolution utilised to correct the issue and any further actions or recommendations to be considered to facilitate opportunities for improvement. These are logged within the ticket and Major Incident (MI) tracker and tracked through to closure.