February 13th, 2023

Introduction

We tried backup and restore using an Amazon EBS snapshot (hereinafter called "EBS snapshot") for mirror disks of EXPRESSCLUSTER X HA cluster built on Amazon Web Services (hereinafter called "AWS").

In this blog, we have introduced backup and restore using an EBS snapshot in EXPRESSCLUSTER X for Windows in the past.

[Reference]
We Tried Backing up and Restoring Mirror Disks with an Amazon EBS Snapshot in Amazon Web Services (Windows)

This time, we introduce backup and restore using EXPRESSCLUSTER X for Linux. Also, we use pre- and post-processing commands for backup/restore, which are supported in EXPRESSCLUSTER X 4.3 and later.

Pre- and post-processing commands for backup/restore make it simple and safe to execute the complex pre- and post-processing required when backup and restore of partitions for mirror/hybrid disks.

[Reference]
Introducing the Function of EXPRESSCLUSTER: Simplify Backup/Restore

Contents

1. HA Cluster Configuration

This time, we use a two-node mirror disk type HA cluster for Linux on AWS as an example.
The environment and EXPRESSCLUSTER version used are as follows:

The HA cluster configuration is as follows:
This time, we use a two-node active-standby cluster configuration of server1 and server2. Application data is shared by placing it on a mirror disk. The server to which the VIP (Virtual IP address) is assigned (the server accessed by the client machine) is the active server, and the other server is the standby server.
Since this is just an example of a general redundancy configuration using mirror disk with EXPRESSCLUSTER, the network configuration does not need to be the same as this time to perform EBS backup/restore. Please change the reading according to your environment as appropriate.

The mirror disk configuration of EXPRESSCLUSTER is as follows:
There are two EBS volumes attached to an EC2 instance for one server: the root device attached when the EC2 instance is created, and the newly created and attached EBS for the mirror disk.

- EC2 instance / EBS volume
  ■Common to server1 (instance ID: i-11111111aaaaaaa) / server2 (instance ID: i-11111111bbbbbbb)

EXPRESSCLUSTER

Please refer to the following for the setting procedure in the AWS environment.

[Reference]
Documentation - Setup Guides

• Linux > Cloud > Amazon Web Services
  > EXPRESSCLUSTER X 5.0 for Linux HA Cluster Configuration Guide for Amazon Web Services

In this procedure, we run the AWS CLI from the client machine when performing operations such as launching an EC2 instance, creating an EBS snapshot, creating an EBS volume from an EBS snapshot, and detaching and attaching an EBS volume. Therefore, create a policy that describes the permissions for these actions and grant it to the client machine.
Add the following permissions:

If you do not use the AWS CLI and operate from the AWS Management Console, you do not need to add the above permissions.

2. Backup Procedure

2.1 Overview

Create a backup of the mirror disk.
There are several backup method procedures, but this time we introduce the procedure "When backing up a mirror disk on the standby server".

Please refer to the following for instructions and details on other backup methods.

[Reference]
Documentation - Manuals

• EXPRESSCLUSTER X 5.0 > EXPRESSCLUSTER X 5.0 for Linux > Maintenance Guide
  > 2 The system maintenance information
    > 2.16 How to back up a mirror/hybrid disk to its disk image

2.2 Stopping a Failover Group

Check that the mirror disk can be synchronized successfully.
Run the following command on the active or the standby server.

After checking the synchronization status, stop the failover group to secure a quiescent point.
Run the following command on the active server.

2.3 Stopping a Mirror Disk Synchronization

Suspend the mirror disk monitor resource to prevent auto mirror recovery from performing.
Run the following commands on the active or the standby server.

Stop the mirror synchronization.
Run the following command on the standby server.

If you want to resume your business immediately, start failover group on the active server at this stage.
Run the following command on the active server.

2.4 Running Pre-processing Command for Backup (Stopping the Standby Server)

Run the following command on the standby server.

2.5 Creating an EBS Snapshot

Take an EBS snapshot of the mirror disk of the standby server.
Run the following command on the client machine.

Check in the AWS Management Console etc. that the EBS snapshot has been created.
Note the snapshot ID of the created EBS snapshot since it will be used later. The snapshot ID can also be obtained by [SnapshotId] in the command execution result.

Since the EBS snapshot is taken asynchronously, after executing to take the EBS snapshot, you may proceed to the next section "2.6 Starting the Standby Server" even while the snapshot is being created. However, it is recommended to secure a quiescent point when creating a snapshot in order to maintain data integrity ("2.2 Stopping a Failover Group" and "2.3 Stopping a Mirror Disk Synchronization" correspond to the procedure for secure a quiescent point).

[Reference]
Amazon EBS snapshots

2.6 Starting the Standby Server

Run the following command on the client machine to start the standby server.

2.7 Running Post-processing Command for Backup (Rebooting the Standby Server)

Run the following command on the standby server.

2.8 Resume a Mirror Disk Synchronization

Resume mirror disk monitor resource on the active server.
Run the following command on the active server.

If the failover group was not started on the active server in "2.3 Stopping a Mirror Disk Synchronization", start it and restart mirror synchronization.
Run the following command on the active server.

When auto mirror recovery is enabled, the differences between the active and the standby mirror disks made during the backup operation are automatically synchronized, and they are restored to normal status. If auto mirror recovery is not executed and the mirror disk does not go back to normal, run the following command on the active or the standby server.

3. Restore Procedure

3.1 Overview

Suppose that data is deleted from the active mirror disk due to an operation error, and data is deleted from both mirror disks because it is synchronized with the standby mirror disk.

Stop the failover group and replace the mirror disk on the active and the standby servers with volumes created from the same backup (EBS snapshot). Since the contents of the active and the standby mirror disks are the same and there is no difference, the full copy process of the mirror disk is not required.

3.2 Changing Mirror Disk Resource Setting

When running the post-processing command for restore, [Execute the initial mirror construction] must be disabled in the mirror disk resource settings (if it is enabled, the post-processing command for restore cannot be run).
Connect to Cluster WebUI and switch to "Config mode". In the properties of the mirror disk resource, click the [Tuning] button on the [Details] tab. If [Execute the initial mirror construction] is enabled on the [Mirror] tab, disable it. After changing the settings, execute [Apply the Configuration File].

3.3 Enabling EBS Fast Snapshot Restore

For EBS volumes that has just been created from an EBS snapshot, the storage blocks must be pulled down from Amazon S3 and written to the volume on first access. This preliminary action takes time and can cause a significant increase in the latency of I/O operations the first time each block is accessed. If no action is taken, this disk I/O delay is likely to degrade service response performance or cause monitoring anomalies due to processing delays when business resumes after a restore.
Please refer to the following for details and how to deal with it.

[Reference]
Initialize Amazon EBS volumes

To avoid disk I/O delays, enable EBS fast snapshot restore for backup (EBS snapshot) before restoring. Note that activation takes 60 minutes per TiB to complete.
If I/O performance is not an issue, this step can be omitted.
Run the following command on the client machine.

Please refer to the following for details about EBS fast snapshot restore.

[Reference]
Amazon EBS fast snapshot restore

3.4 Creating EBS Volumes

Create EBS volumes for restore from a backup (EBS snapshot).
Create an EBS volume for each mirror disk of the active and the standby server from the EBS snapshot taken in "2.5 Create an EBS snapshot". At this time, the Availability Zone (hereinafter called "AZ") to be created should be the same AZ as the server to which EBS volume will be attached.
Run the following commands on the client machine.

3.5 Disabling EBS Fast Snapshot Restore

While EBS fast snapshot restore is enabled, you are charged a recurring cost.
Therefore, if you have enabled EBS fast snapshot restore, disable it.
Run the following command on the client machine.

3.6 Stopping a Failover Group

Stop the failover group.
Run the following command on the active server.

3.7 Running Pre-processing Command for Restore (Stopping the Active/Standby server)

Run the following command on both the active and the standby servers.

3.8 Detaching Old EBS Volumes

Detach the EBS volumes for the mirror disk on each server.
Run the following commands on the client machine.

3.9 Attaching New EBS Volumes

Attach the EBS volumes created in "3.4 Creating EBS Volumes" to each server.
Run the following command on the client machine.

3.10 Starting the Active/Standby Server

Run the following commands on the client machine to start both the active and the standby servers.

3.11 Checking the Mirror Disk Setting

After starting both the active and the standby servers, check the device file name of the restored the cluster partition and the data partition with the lsblk command.

If the device file name set in the mirror disk resource has changed, change the mirror disk settings. Connect to Cluster WebUI and switch to "Config mode". In the properties of the mirror disk resource, enter the correct device file name in [Data Partition Device Name] and [Cluster Partition Device Name] on the [Details] tab.
After entering, execute [Apply the Configuration File].

[Reference]
Points to Note When Building a Mirror Disk Type Cluster on Linux in a Cloud Environment

3.12 Running Post-processing Command for Restore (Rebooting the Active/Standby server)

Run the following command on both the active and the standby servers.

3.13 Checking the Result of Restore

Check that EXPRESSCLUSTER X starts normally and that the status of the mirror disk resource is normal.
Run the following command on the active or the standby server.

Conclusion

In this article, we introduced the procedure for backup and restore mirror disk on AWS using pre- and post-processing commands for backup/restore, which are supported in EXPRESSCLUSTER X 4.3 and later.
Since mirror disk often store important business data,it is essential for operation to prepare backup procedures in case data is destroyed or deleted due to failures or work errors. For actual operation, please also refer to the following for backup and restore procedure.

[Reference]
Documentation - Manuals

• EXPRESSCLUSTER X 5.0 > EXPRESSCLUSTER X 5.0 for Linux > Maintenance Guide
  > 2 The system maintenance information
    > 2.16 How to back up a mirror/hybrid disk to its disk image
    > 2.17 How to restore the mirror/hybrid disk from the disk image

trial module