DR Testing and Remediation

Customer Challenge

After the implementation of a new hybrid cloud platform to support 20+ countries and 500+ business critical applications the client retained Citihub Consulting to mobilise, plan and manage the first annual DR test. The platform is physically hosted in several European locations and integrated multiple external and internal service providers.

The test was subject to intense business scrutiny on the performance of the platform and the service delivery organisation. Findings needed to be documented in detail and contractually enforceable with service providers.

How Citihub Consulting Helped

Citihub mobilised a small team that:

• Critically assessed DR policies, procedures and documentation to ensure that they reflected the ‘new’ recovery position after migration to the Hybrid Cloud;
• Identified gaps or deficiencies in the recovery plans and worked with service providers to remediate these;
• Conducted a set of progressively more mature table-top DR exercises to test documentation, procedures and the knowledge of key individuals;
• Built disaster scenarios, syndicated and agreed a set of tests with business stakeholders;
• Performed a deep-dive technical assessment of the platform and its readiness for the test so as to reduce the risk of a business impacting outage;
• Mobilised the test resources (service providers, internal IT, IT security, application teams, and business testing resources – 300+ staff in total) and coordinated the development of integrated test plans;
• Choreographed and monitored the event in real time and intervened where necessary to ensure success;
• Measured the performance of the DR test using KPIs and SLAs;
• Executed all business facing communications both pre-and-post the test;
• Documented all lessons learned along with technology, skill, documentation and process deficiencies;
• Produced remediation and improvement recommendations covering technology infrastructure, outsourced commercial arrangements, and service delivery.

Customer Benefits

Citihub Consulting’s approach delivered a structured, well planned and measurable test of the end-to-end IT solution and applications.

Recovery steps were documented in a timeline to give visibility to the business and to inform future investment decisions for priority services. Then remediation and improvement actions were prioritised so that traceable work packages could be developed and mobilised. All high probability and high impact risks were quickly addressed by implementing technology improvements or process changes.

The testing methodology took a holistic approach by examining the performance of all aspects of the IT platform including: network, data centres, servers, storage, databases, applications and IT security infrastructure. This approach was specifically designed to ensure a universal test that could identify any stacked failures resulting from technology and process dependencies.

The test strategy, governance, preparation guidelines and testing plans now form the basis of a repeatable annual DR test. This approach has now been adopted globally.