Global Site
Displaying present location in the site.
November 30th, 2022
Machine translation is used partially for this article. See the Japanese version for the original article.
Introduction
For the mirror disk type HA cluster built with EXPRESSCLUSTER X on Amazon Web Services (hereinafter called "AWS"), assuming a situation where business continuity becomes difficult due to a failure of the region you usually use, we tried a method of restoring the backed up cluster to another region. This method can also be used as a means of disaster recovery (DR) in the event of a large-scale disaster that affects the entire region.
There are various ways to back up AWS resources, such as the functions provided by AWS and the third-party tools and services, but this time we use only the functions provided by AWS.
This article describes the backup procedure of the overall backup and restore procedures. For instructions of restore, refer to here.
Contents
1. How to Back up EC2 Instances
We will introduce "EBS snapshots" and "Amazon Machine Images (hereinafter called "AMI")", which are often used as backup methods for EC2 instances.
EBS snapshots
EBS snapshot is a backup method that creates a snapshot of the EBS volume that contains the EC2 instance data, and it is stored in Amazon S3. We can restore EC2 instance data to the data at the time of snapshot acquisition by creating (restoring) a new EBS volume from an EBS snapshot and replacing it with a volume attached to an existing EC2 instance.
EBS snapshots do not contain the configuration information required to launch EC2 instances. Therefore, if you want to create a new EC2 instance from an EBS snapshot, there are multiple steps: At first, create a new image (AMI) from the root volume snapshot, and then create a new EC2 instance from the image. In addition, if there are multiple EBS volumes attached to the EC2 instance, you need to perform the task of "creating a new volume from a snapshot and attaching it to a new EC2 instance" for each volume.
AMI
In addition to EBS snapshots, AMI is a backup method that also includes configuration information required to launch EC2 instances. If there are multiple EBS volumes attached to an EC2 instance, the snapshots of all volumes are included in a single AMI. AMI data is stored in Amazon S3 same as EBS snapshots. By creating (restoring) a new EC2 instance from an AMI, you can restore an instance with data at the time the AMI was created.
In this article, it is assumed that a new server will be created in the restore destination region using the backed-up EC2 instance data, so we will adapt an AMI that can easily create EC2 instances as a backup method.
2. HA Cluster Configuration
This time, we will use a 2 nodes mirror disk type HA cluster built in the N. Virginia region, and use the AMI created as a backup to build an equivalent HA cluster in the Oregon region. The combinations of environments evaluated are as follows:
- * For the combinations of environments with EXPRESSCLUSTER X 4.2 or before, refer to the Japanese version of this article.
AWS Region | N. Virginia (us-east-1) and Oregon (us-west-2) |
Server OS and EXPRESSCLUSTER X version | Windows - EXPRESSCLUSTER X 5.0 + Windows Server 2022 Linux - EXPRESSCLUSTER X 5.0 + Red Hat Enterprise Linux 8.4 |
The HA cluster configurations are follows. Focus on the left side of the diagram (the cluster in the N. Virginia region).
In a 2 nodes Active-Standby cluster configuration of server01 and server02, application data written to the mirror disk on the active side is mirrored to the EBS volume on the standby side. The side for the VIP (virtual IP address) and virtual hostname (Route 53 DNS record), the server accessed by the client, is the active server, and the other side is the standby server.
Since this is just an example of a general redundant configuration using mirror disk with EXPRESSCLUSTER, the network configuration does not need to be the same as the example configuration to perform backup and restore of EC2 instances. Please change to read according to your environment as appropriate.
The configuration of the mirror disk of EXPRESSCLUSTER is as follows.
The number of block devices attached to each server instance is three: one EBS for the root volume prepared from the beginning when the instances are created, and two EBSs added for the mirror disk.
The following settings are recommended when creating the spaces for the cluster partition and data partition in EBS for mirror disks.
- Windows: Specify GPT as the partition format
- Linux: Create partitions using LVM (Logical Volume Manager)
In an AWS environment, the EBS device names attached to the instance might not match before and after the restore. If the restored device name is different from the one before the restore, you will need to change the mirror disk resource settings in EXPRESSCLUSTER. Therefore, by setting the above, we take measures so that it is not necessary to change the settings of EXPRESSCLUSTER after restoration.
- Server Storage Breakdown (Common to server01/server02)
Device names (example) | Usage | Drive letters (Windows) or Device file names (Linux) |
/dev/sda1 | System area of the OS | C: or /dev/sda1 |
xvdf or /dev/sdf | Mirror disk (md1) - Cluster partition - Data partition |
D: or /dev/clp_md1/cp E: or /dev/clp_md1/dp |
xvdg or /dev/sdg | Mirror disk (md2) - Cluster partition - Data partition |
F: or /dev/clp_md2/cp G: or /dev/clp_md2/dp |
- EXPRESSCLUSTER Resource Settings
- Failover group (failover)
- ■Mirror disk resource (md1)
- - Cluster partition: D: or /dev/clp_md1/cp
- - Data partition: E: or /dev/clp_md1/dp
- ■Mirror disk resource (md2)
- - Cluster partition: F: or /dev/clp_md2/cp
- - Data partition: G: or /dev/clp_md2/dp
- ■AWS Virtual IP resource (awsvip)
- - IP Address: 172.16.0.1 (This will be changed when restoring in the Oregon region)
- ■AWS DNS resource (awsdns)
- - Resource Record Set Name: virginia.expresscluster.local. (This will be changed when restoring in the Oregon region)
- Monitor Resources
- * Omitted because they are automatically registered by adding the above group resources.
- If you have not changed the hostname since you created the EC2 instances, the hostname of each server restored from the AMI changes to a different hostname from the backup source. When the hostname changes, the steps of updating EXPRESSCLUSTER configuration information increases during the restore procedure. Therefore, this time, the hostname is set in advance so that it is fixed.
- This article is not intended to create reusable AMIs (Golden Images). Therefore, we have not run Sysprep, which removes instance security identifiers (SID), computer names, etc. from instances in Windows environments before creating the AMI.
- Credentials are required to log in to the server to be restored to the Oregon region side. If your server is a Linux environment, import an existing key pair in the N. Virginia region into the Oregon region. If the server is a Windows environment, note of the server login password so as not to forget it.
3. Backup Procedure
3.1 Pre-processing of Backup
Check that the mirror disk can be synchronized successfully.
Run the following command on server01 or server02.
(Each subsequent command is run as a user with administrator/root privileges.)
Mirror Status: Normal
md1 server01 server02
------------------------------------------------------------
Mirror Color GREEN GREEN
(Omitted)
After checking the synchronization status, stop the failover group. On the active server where the failover group is running, run the following command:
Execution example
Command succeeded.
The steps vary depending on your version of EXPRESSCLUSTER X.
■For X 4.3 or later
Run the clpbackup command. The server automatically shuts down when the command completes.
Execution example (Windows)
clpbackup.bat : Beginning backup-mode.
Command succeeded.
clpbackup.bat : Changing the setting of cluster services to Manual Startup.
clpbackup.bat : Shutting down...
Command succeeded.
clpbackup.bat : Command succeeded.
clpbackup.sh : Beginning backup-mode.
All backup flags have set to <ON>.
clpbackup.sh : Changing the setting of cluster services to Manual Startup.
clpbackup.sh : Shutting down...
Command succeeded.
clpbackup.sh : Command succeeded.
Use the clpsvcctrl command to change the prescribed EXPRESSCLUSTER service to manual startup.
Execution example (Windows)
> clpdown
# clpdown
Use the command provided by the OS to change the prescribed EXPRESSCLUSTER service to manual startup. Then shut down the server by running clpdown command.
Execution example (Windows)
[SC] ChangeServiceConfig SUCCESS
> clpdown
Removed /etc/systemd/system/multi-user.target.wants/clusterpro.service.
# systemctl disable clusterpro_md
Removed /etc/systemd/system/multi-user.target.wants/clusterpro_md.service.
# clpdown
3.2 Backing up
Create backups of each EBS by creating AMIs for server01, server02.
In the AWS Management Console, select the EC2 instance of server01 or server02, and then click [Create image].
Confirm that all volumes to be backed up are included on the [Create image] screen, and when you have finished entering the image name and image description, click [Create Image].
When the image you created appears in the list of Amazon Machine Images (AMIs) in the AWS Management Console, you are finished creating the backup.
3.3 Post-processing of Backup
Start EC2 instances of server01 and server02.
After starting the instances, perform the following steps on both server01 and server02 to enable operation as an HA cluster.
The steps vary depending on your version of EXPRESSCLUSTER X.
■ For X 4.3 or later
Run the clpbackup command. The server restarts automatically when the command completes.
clpbackup.bat : Ending backup-mode.
Command succeeded.
clpbackup.bat : Changing the setting of cluster services to Auto Startup.
clpbackup.bat : Rebooting...
clpbackup.sh : Starting Mirror Agent.
Command succeeded.
clpbackup.sh : Ending backup-mode.
All backup flags have set to <OFF>.
clpbackup.sh : Changing the setting of cluster services to Auto Startup.
clpbackup.sh : Stopping Mirror Agent.
Command succeeded.
clpbackup.sh : Rebooting...
Use the clpsvcctrl command to change the predetermined EXPRESSCLUSTER service to auto-start. Then restart the server by running clpdown command.
Execution example (Windows)
> clpdown -r
# clpdown -r
Use the command provided by the OS to change the predetermined EXPRESSCLUSTER service to auto-start. Then restart the server by running shutdown/reboot command.
Execution example (Windows)
[SC] ChangeServiceConfig SUCCESS
> shutdown -r -t 0
Created symlink /etc/systemd/system/multi-user.target.wants/clusterpro.service → /usr/lib/systemd/system/clusterpro.service.
# systemctl enable clusterpro_md
Created symlink /etc/systemd/system/multi-user.target.wants/clusterpro_md.service → /usr/lib/systemd/system/clusterpro_md.service.
# reboot
3.4 Copying Images
Copy the created AMIs (backup) to another region.
In the AWS Management Console, select the image in the Amazon Machine Images (AMIs) list, and then click [Copy AMI].
Select "US West (Oregon)" for [Destination Region] on the [Copy AMI] screen, and then click [Copy AMI].
Change the destination region of the AWS Management Console to the Oregon region where you copied the AMIs, and then list your Amazon Machine Images (AMIs). Check that the AMIs you copied are displayed.
Conclusion
In this article, we explained the procedure for backing up EC2 instances using AMI. If you copy the created AMIs to another region in advance, when the entire region you usually use does not work, you can continue business by restoring the cluster to another region. For restore procedures, refer to here.
If you consider introducing the configuration described in this article, you can perform a validation with the trial module of EXPRESSCLUSTER. Please do not hesitate to contact us if you have any questions.