TABLE OF CONTENTS
Scenario: Unexpected data loss within DomaCare
Summary
Invian Oy maintains a backup of all data in the DomaCare system. All customer data is backed up every day in its entirety and smaller backups are maintained of frequently problematic data, so that in the case of accidental deletion or mishandling of data Invian Oy is able to recover it. Internal processes are well established and practised for data recovery operations of commonly mishandled data.
DomaCare has safeguards in place to prevent malicious users from destroying or maliciously altering customer data, as well as safeguards to prevent accidental deletion or overriding of critical data. Customer related documents can be versioned and version history be made read only in addition to backups to further ensure data integrity.
Scenario: Total loss of hosted server infrastructure
Summary
This document describes the actions Invian Oy will take to ensure the continuity and recovery of DomaCare in the unlikely event of total and long lasting outage of the main hosted server infrastructure.
DomaCare is hosted in data centres located in Finland. DomaCare has only experienced a handful of short partial outages and degraded performance incidents attributable to the hosting provider, and the selected hosting provider has been deemed extremely reliable.
The data in DomaCare is also backed up to an independent external server to ensure data integrity and availability. Using a combination of backup methods the data in DomaCare is never subject to a single point of failure. Invian also has additional security and infrastructure support resources available from its parent company Visma Oy.
Severity: Very High
Likelihood: Very Unlikely
Estimated Effect: Total service loss for multiple days
Recovery Plan
In the event of a total failure of DomaCare infrastructure the following steps will be taken:
Initial Response
Notifying the infrastructure and security teams inside Invian
Establishing communications with main hosting provider
Notifying all customers
Notifying Visma security and infrastructure teams
Assessing Options
Establishing the extent of the service failure
Preparing a work plan and assigning work
Disaster Recovery
Recovering the external backups of the system
Establishing new server infrastructure
Configuring the new server infrastructure and applying the backup on it
Following is a description of each step.
Initial Response
A total infrastructure failure would be detected by the automatic monitoring in place in various parts of the infrastructure. Automatic monitoring will automatically alert both the infrastructure and security teams at Invian. From here, the infrastructure team will take the necessary steps to start assessing the situation and notifying the other parts of Invian. The infrastructure team is responsible for recovering the service. As part of the initial response, the infrastructure team will contact the emergency contacts for Invian at the hosting provider and will start establishing the extent of the service outage. Other parties inside Invian will start notifying customers of the service outage. The incident will also be escalated to relevant teams inside Visma for further support.
The automatic infrastructure monitoring detects many types of failures and alerts the infrastructure team automatically. The metrics tracked include but are not limited to memory and storage use, connectivity, and user load on the servers. These serve as an early warning system and help prevent outages.
Assessing Options
Once the extent of the service outage is established the infrastructure team will prepare a plan for ensuring the continuity of DomaCare. As the server infrastructure and current environment is deemed unrecoverable, the infrastructure team will create a work plan for setting up DomaCare using an alternative hosting provider. However, if the hosting provider is not down in its entirety, then DomaCare will be moved to another datacenter inside the European Economic Area.
The infrastructure team has a working prototype for a minimal viable deployment of DomaCare that can be set up at will. Setting up this environment has been practised during external vulnerability and security audits when a dedicated environment is required. The next disaster recovery exercise is being planned for Fall 2023.
Disaster Recovery
Once the scope of the outage is identified and confirmed to be severe, the infrastructure team will begin the system recovery process. All data in DomaCare is restored from the external system backup and the system can be recovered on a timescale of days. The expected outage in this scenario would not exceed one week. Some minor data loss is to be expected as the backup for this type of failure does not allow point-in-time recovery. Customers are informed promptly when the service is operational again. In the event of delays or unexpected challenges, the DomaCare team is able to provide critical information directly to the customer, such as shifts, bookings, and customer medication reports.
The steps required for recovering the full DomaCare system are complex and dependent on the extent of the service failure and are not covered here in length.