Displaying present location in the site.

Backup and Restore Method for an HA Cluster across Regions Using AWS AMI: Backup

EXPRESSCLUSTER Official Blog

November 30th, 2022

Machine translation is used partially for this article. See the Japanese version for the original article.

Introduction

For the mirror disk type HA cluster built with EXPRESSCLUSTER X on Amazon Web Services (hereinafter called "AWS"), assuming a situation where business continuity becomes difficult due to a failure of the region you usually use, we tried a method of restoring the backed up cluster to another region. This method can also be used as a means of disaster recovery (DR) in the event of a large-scale disaster that affects the entire region.

There are various ways to back up AWS resources, such as the functions provided by AWS and the third-party tools and services, but this time we use only the functions provided by AWS.

This article describes the backup procedure of the overall backup and restore procedures. For instructions of restore, refer to popuphere.

Contents

1. How to Back up EC2 Instances

We will introduce "EBS snapshots" and "Amazon Machine Images (hereinafter called "AMI")", which are often used as backup methods for EC2 instances.

EBS snapshots
EBS snapshot is a backup method that creates a snapshot of the EBS volume that contains the EC2 instance data, and it is stored in Amazon S3. We can restore EC2 instance data to the data at the time of snapshot acquisition by creating (restoring) a new EBS volume from an EBS snapshot and replacing it with a volume attached to an existing EC2 instance.
EBS snapshots do not contain the configuration information required to launch EC2 instances. Therefore, if you want to create a new EC2 instance from an EBS snapshot, there are multiple steps: At first, create a new image (AMI) from the root volume snapshot, and then create a new EC2 instance from the image. In addition, if there are multiple EBS volumes attached to the EC2 instance, you need to perform the task of "creating a new volume from a snapshot and attaching it to a new EC2 instance" for each volume.

AMI
In addition to EBS snapshots, AMI is a backup method that also includes configuration information required to launch EC2 instances. If there are multiple EBS volumes attached to an EC2 instance, the snapshots of all volumes are included in a single AMI. AMI data is stored in Amazon S3 same as EBS snapshots. By creating (restoring) a new EC2 instance from an AMI, you can restore an instance with data at the time the AMI was created.

In this article, it is assumed that a new server will be created in the restore destination region using the backed-up EC2 instance data, so we will adapt an AMI that can easily create EC2 instances as a backup method.

2. HA Cluster Configuration

This time, we will use a 2 nodes mirror disk type HA cluster built in the N. Virginia region, and use the AMI created as a backup to build an equivalent HA cluster in the Oregon region. The combinations of environments evaluated are as follows:

  • * For the combinations of environments with EXPRESSCLUSTER X 4.2 or before, refer to the Japanese version of this article.
AWS Region N. Virginia (us-east-1) and Oregon (us-west-2)
Server OS and EXPRESSCLUSTER X version Windows
- EXPRESSCLUSTER X 5.0 + Windows Server 2022

Linux
- EXPRESSCLUSTER X 5.0 + Red Hat Enterprise Linux 8.4

The HA cluster configurations are follows. Focus on the left side of the diagram (the cluster in the N. Virginia region).
In a 2 nodes Active-Standby cluster configuration of server01 and server02, application data written to the mirror disk on the active side is mirrored to the EBS volume on the standby side. The side for the VIP (virtual IP address) and virtual hostname (Route 53 DNS record), the server accessed by the client, is the active server, and the other side is the standby server.
Since this is just an example of a general redundant configuration using mirror disk with EXPRESSCLUSTER, the network configuration does not need to be the same as the example configuration to perform backup and restore of EC2 instances. Please change to read according to your environment as appropriate.

The configuration of the mirror disk of EXPRESSCLUSTER is as follows.
The number of block devices attached to each server instance is three: one EBS for the root volume prepared from the beginning when the instances are created, and two EBSs added for the mirror disk.
The following settings are recommended when creating the spaces for the cluster partition and data partition in EBS for mirror disks.
- Windows: Specify GPT as the partition format
- Linux: Create partitions using LVM (Logical Volume Manager)
In an AWS environment, the EBS device names attached to the instance might not match before and after the restore. If the restored device name is different from the one before the restore, you will need to change the mirror disk resource settings in EXPRESSCLUSTER. Therefore, by setting the above, we take measures so that it is not necessary to change the settings of EXPRESSCLUSTER after restoration.

- Server Storage Breakdown (Common to server01/server02)

Device names (example) Usage Drive letters (Windows) or Device file names (Linux)
/dev/sda1 System area of the OS C: or /dev/sda1
xvdf or /dev/sdf Mirror disk (md1)
- Cluster partition
- Data partition
D: or /dev/clp_md1/cp
E: or /dev/clp_md1/dp
xvdg or /dev/sdg Mirror disk (md2)
- Cluster partition
- Data partition
F: or /dev/clp_md2/cp
G: or /dev/clp_md2/dp

- EXPRESSCLUSTER Resource Settings

  • Failover group (failover)

  • Mirror disk resource (md1)
  • - Cluster partition: D: or /dev/clp_md1/cp
  • - Data partition: E: or /dev/clp_md1/dp
  • Mirror disk resource (md2)
  • - Cluster partition: F: or /dev/clp_md2/cp
  • - Data partition: G: or /dev/clp_md2/dp
  • AWS Virtual IP resource (awsvip)
  • - IP Address: 172.16.0.1 (This will be changed when restoring in the Oregon region)
  • AWS DNS resource (awsdns)
  • - Resource Record Set Name: virginia.expresscluster.local. (This will be changed when restoring in the Oregon region)
  • Monitor Resources

  • * Omitted because they are automatically registered by adding the above group resources.
- Other precautions
  • If you have not changed the hostname since you created the EC2 instances, the hostname of each server restored from the AMI changes to a different hostname from the backup source. When the hostname changes, the steps of updating EXPRESSCLUSTER configuration information increases during the restore procedure. Therefore, this time, the hostname is set in advance so that it is fixed.
  • This article is not intended to create reusable AMIs (Golden Images). Therefore, we have not run Sysprep, which removes instance security identifiers (SID), computer names, etc. from instances in Windows environments before creating the AMI.
  • Credentials are required to log in to the server to be restored to the Oregon region side. If your server is a Linux environment, import an existing key pair in the N. Virginia region into the Oregon region. If the server is a Windows environment, note of the server login password so as not to forget it.

3. Backup Procedure

3.1 Pre-processing of Backup

Check that the mirror disk can be synchronized successfully.
Run the following command on server01 or server02.
(Each subsequent command is run as a user with administrator/root privileges.)

Execution example
# clpmdstat --mirror md1

  Mirror Status: Normal

  md1                    server01             server02
  ------------------------------------------------------------
  Mirror Color        GREEN               GREEN
(Omitted)
If the Mirror Color is GREEN on both server01 and server02, the mirror disk is successfully synchronized.
After checking the synchronization status, stop the failover group. On the active server where the failover group is running, run the following command:

Execution example
# clpgrp -t failover
Command succeeded.
Then perform the following steps on both server01 and server02:
The steps vary depending on your version of EXPRESSCLUSTER X.

For X 4.3 or later
Run the clpbackup command. The server automatically shuts down when the command completes.

Execution example (Windows)
> clpbackup --pre
clpbackup.bat : Beginning backup-mode.
Command succeeded.
clpbackup.bat : Changing the setting of cluster services to Manual Startup.
clpbackup.bat : Shutting down...
Command succeeded.
clpbackup.bat : Command succeeded.
Execution example (Linux)
# clpbackup.sh --pre
clpbackup.sh : Beginning backup-mode.
All backup flags have set to <ON>.
clpbackup.sh : Changing the setting of cluster services to Manual Startup.
clpbackup.sh : Shutting down...
Command succeeded.
clpbackup.sh : Command succeeded.
For X 4.2
Use the clpsvcctrl command to change the prescribed EXPRESSCLUSTER service to manual startup.

Execution example (Windows)
> clpsvcctrl --disable core
> clpdown
Execution example (Linux)
# clpsvcctrl.sh --disable core
# clpdown
For X 4.1
Use the command provided by the OS to change the prescribed EXPRESSCLUSTER service to manual startup. Then shut down the server by running clpdown command.

Execution example (Windows)
> sc.exe config clppm start= demand
[SC] ChangeServiceConfig SUCCESS
> clpdown
Execution example (Linux)
# systemctl disable clusterpro
Removed /etc/systemd/system/multi-user.target.wants/clusterpro.service.
# systemctl disable clusterpro_md
Removed /etc/systemd/system/multi-user.target.wants/clusterpro_md.service.
# clpdown

3.2 Backing up

Create backups of each EBS by creating AMIs for server01, server02.
In the AWS Management Console, select the EC2 instance of server01 or server02, and then click [Create image].

Confirm that all volumes to be backed up are included on the [Create image] screen, and when you have finished entering the image name and image description, click [Create Image].

When the image you created appears in the list of Amazon Machine Images (AMIs) in the AWS Management Console, you are finished creating the backup.

3.3 Post-processing of Backup

Start EC2 instances of server01 and server02.

After starting the instances, perform the following steps on both server01 and server02 to enable operation as an HA cluster.
The steps vary depending on your version of EXPRESSCLUSTER X.

For X 4.3 or later
Run the clpbackup command. The server restarts automatically when the command completes.

Execution example (Windows)
> clpbackup --post
clpbackup.bat : Ending backup-mode.
Command succeeded.
clpbackup.bat : Changing the setting of cluster services to Auto Startup.
clpbackup.bat : Rebooting...
Execution example (Linux)
# clpbackup.sh --post
clpbackup.sh : Starting Mirror Agent.
Command succeeded.
clpbackup.sh : Ending backup-mode.
All backup flags have set to <OFF>.
clpbackup.sh : Changing the setting of cluster services to Auto Startup.
clpbackup.sh : Stopping Mirror Agent.
Command succeeded.
clpbackup.sh : Rebooting...
For X 4.2
Use the clpsvcctrl command to change the predetermined EXPRESSCLUSTER service to auto-start. Then restart the server by running clpdown command.

Execution example (Windows)
> clpsvcctrl --enable core
> clpdown -r
Execution example (Linux)
# clpsvcctrl.sh --enable core
# clpdown -r
For X 4.1
Use the command provided by the OS to change the predetermined EXPRESSCLUSTER service to auto-start. Then restart the server by running shutdown/reboot command.

Execution example (Windows)
> sc.exe config clppm start= auto
[SC] ChangeServiceConfig SUCCESS
> shutdown -r -t 0
Execution example (Linux)
# systemctl enable clusterpro
Created symlink /etc/systemd/system/multi-user.target.wants/clusterpro.service → /usr/lib/systemd/system/clusterpro.service.
# systemctl enable clusterpro_md
Created symlink /etc/systemd/system/multi-user.target.wants/clusterpro_md.service → /usr/lib/systemd/system/clusterpro_md.service.
# reboot

3.4 Copying Images

Copy the created AMIs (backup) to another region.
In the AWS Management Console, select the image in the Amazon Machine Images (AMIs) list, and then click [Copy AMI].

Select "US West (Oregon)" for [Destination Region] on the [Copy AMI] screen, and then click [Copy AMI].

Console5

Change the destination region of the AWS Management Console to the Oregon region where you copied the AMIs, and then list your Amazon Machine Images (AMIs). Check that the AMIs you copied are displayed.

Conclusion

In this article, we explained the procedure for backing up EC2 instances using AMI. If you copy the created AMIs to another region in advance, when the entire region you usually use does not work, you can continue business by restoring the cluster to another region. For restore procedures, refer to popuphere.

If you consider introducing the configuration described in this article, you can perform a validation with the popuptrial module of EXPRESSCLUSTER. Please do not hesitate to contact us if you have any questions.