We registered "Main" and "Sub" as server groups for EXPRESSCLUSTER, adding server1 and server2 in the N. Virginia region to "Main," and server3 in the Oregon region to "Sub." With this setup, you will basically be operating an HA cluster across servers registered in the "Main" server group. In the event of a regional failure such as the N. Virginia region going down, failover occurs to the Oregon region, allowing business operations to continue with server3 registered in the "Sub" server group.

Amazon EFS is mounted with NFS in the active system. NFS mounting is not performed at the OS boot on each node because, as explained in "1. About Amazon EFS Replication" the file system ID may change during failover.

Additionally, a client is prepared in the Ohio region for the purpose of verifying operations such as connecting to the application and Cluster WebUI.

Furthermore, switching between the active/standby systems can be done not only through DNS name control but also via VIP control. However, to enable access via VIP from outside the VPC, AWS Transit Gateway is required instead of VPC peering connection. Please refer to the article below for an HA cluster configuration using VIP control with AWS Transit Gateway.

This article focuses on the features of Amazon EFS and replication, so the network partition resolution is not covered. Therefore, please refer to the articles below for practical operations.

A file system is created in the N. Virginia region. On the Amazon EFS file system creation screen, click the "Customize" button to proceed to the detailed settings screen.

On the "File system settings" screen, enter the following values. Other items are optional.

Click the "Next" button.
On the "Network access" screen, enter the following settings to create the mount targets in the private subnet of the VPC where the server instance is located.

Click the "Next" button.
On the "File system policy" screen, click the "Next" button without making any changes.
On the "Review and create" screen, check the contents and click the "Create" button.

In the Amazon EFS file system list screen, select the created file system (DRTest-EFS1) and click the "View details" button.

Click the "Replication" tab located at the bottom of the file system details screen. Click the "Create replication" button.

In the "Create replication" screen, enter the following information. Other fields are optional.

Clicking the "Create replication" button returns to the DRTest-EFS1 file system detail screen.
Ensure that the "Replication state" at the bottom of the "Replication" tab screen is "Enabled" and click the link for the "File system ID" (fs-xxxxxxxxxxxxxxxxx) displayed in the "Destination" row.

Switch to the Oregon Region, where the details screen of the destination file system (replica of the file system) appears. Click the "Tags" tab at the bottom of the screen and add a Name tag with the same name as the file system created in the N. Virginia Region.

Next, click the "Network" tab.
Click the "Create mount target" button.

On the "Network" screen, enter the following settings (ensure the mount target is created in the private subnet of the VPC where the server instance is located).

Click the "Save" button.

As mentioned in "1. About Amazon EFS Replication," the replication settings created in this section will be deleted during cross-region failover.
Therefore, it is necessary to reconfigure the replication settings before executing failback (failover from Oregon region to N. Virginia region). Since the input values differ slightly, refer to the procedure in "4.3 Preparation for Failback."

When mounting Amazon EFS to an EC2 instance, using the Amazon EFS Mount Helper is recommended. Utilizing the Amazon EFS Mount Helper supports encryption of data in transit and ensures the use of recommended mount options for Amazon EFS. For instructions on how to utilize the Amazon EFS Mount Helper, please refer to the following.

For those not using the Amazon EFS mount helper, please refer to the following.

Create the directory /mnt/efs on each server in the HA cluster and set the mount point of Amazon EFS to /mnt/efs.

Verify on each server that the created file system and its replica can be mounted.

After confirming successful mounting, a directory for the monitored files of the disk monitor resource (diskw), to be added later in "3.2.4 Registration of Monitoring Resources," should be created at this time.

Be sure to unmount the mounted file system.

Create a hosted zone in Amazon Route 53 for the DNS name control method. This article discusses the creation of a private hosted zone to enable name resolution in each VPC.

After creating a hosted zone, create a DNS record to allow the client to access drtest.example.com.

Next, install MinIO on servers in an HA cluster configuration. Refer to the following for installation instructions. Any location can be specified as the installation destination for MinIO. This time, install it in "/work" created under the root directory.

The configuration of this article requires the installation of Python and AWS CLI for use with EXEC resources and AWS DNS resources. Please refer to the following for the operational environments and installation methods of Python and AWS CLI.

In addition, to allow operations of Amazon EFS from the script, the IAM policy requires permission for the following actions in addition to those listed in the above construction guide.

This verification utilizes EXPRESSCLUSTER X 5.2 (internal version Linux: 5.2.1-1). Create server groups separately for the N. Virginia and Oregon regions, and mount the Amazon EFS file system on the active server.

The overview of the EXPRESSCLUSTER configuration (group resources, monitor resources) in this article is as follows.

In "Server Common Properties" define server groups as follows. Register the servers in the N. Virginia region (server1, server2) in the Main group, and the server in the Oregon region (server3) in the Sub group.

When creating a failover group, select "Use server group settings."

Click the "Next" button.
Add "Main" and "Sub" to the "Server Groups that can run the Group."

Click the "Next" button.
For "Failover Attributes," under "Automatic Failover," select "Prioritize server group failover policy." Then select "Enable only manual failover among the server groups."

Click the "Next" button to proceed with the registration of the group resources.

Add the following group resources to the failover group.

■ AWS DNS Resource (awsdns)

Connection switching is achieved by updating the value of the A record for drtest.example.com, registered in Amazon Route 53, to the IP address of the active server. For detailed configuration instructions, please refer to the following.

■ EXEC Resource (exec-EFS)

The EXEC resource (exec-EFS) performs mounting and unmounting of Amazon EFS file systems. In case of failover across regions, when mounting a file system that was a replica, replication settings should also be deleted prior to mounting.
Specifically, the following operations are performed in the start script (start.sh):

The start script (start.sh) for exec-EFS is as follows.
The values of the variables at the beginning should be changed according to the environment.
The variable "USE_EFS_MOUNT_HELPER" should be set to "yes" if the helper is installed in "3.1.5 Installing the Amazon EFS Mount Helper" and "no" if not using the helper.
"EFS_NAME" should specify the file system name designated in "3.1.3 Creating Amazon EFS" (here, DRTest-EFS1).
"MOUNT_POINT" specifies the mount point of the file system.
"LOOP_COUNT" and "SLEEP_SEC" are specified for the waiting loop counter and wait seconds for replication setting deletion.
"AWS_CLI" should specify the absolute path to the aws command of the AWS CLI installed in "3.1.8 AWS CLI Configuration."

The stop script for exec-EFS (stop.sh) is as follows. The variable "MOUNT_POINT" specifies the mount point of the file system.

■ EXEC Resource (exec-MinIO)

The EXEC resource (exec-MinIO) manages the start and stop of MinIO. The start script (start.sh) and stop script (stop.sh) are as follows.

Start script (start.sh)
The variable "MOUNT_POINT" specifies the mount point of the file system.

Stop script (stop.sh)

In the "Details" tab of the EXEC resource (exec-MinIO) properties, click the "Tuning" button and change the "Start Script" from "Synchronous" to "Asynchronous."

Register the following monitoring resources.

■ AWS DNS Monitor Resource (awsdnsw)

The AWS DNS monitor resource is automatically added when an AWS DNS resource is added. No specific configuration changes are necessary.

■ Custom Monitor Resource (genw-EFS)

The custom monitor resource (genw-EFS) checks if the Amazon EFS file system is mounted at /mnt/efs. The monitoring script (genw.sh) is as follows.
The variable "MOUNT_POINT" specifies the mount point of the file system.

■ Disk Monitor Resource (diskw)

The disk monitor resource observes whether the Amazon EFS file system is writable. Select "Active" for monitoring timing, as it is only necessary when the Amazon EFS file system is mounted.

■ PID Monitor Resource (pidw-MinIO)

The MinIO service is monitored using the PID monitor resource (pidw-MinIO). The target resource specifies the EXEC resource (e.g., exec-MinIO) that controls MinIO.

The operation check can be performed by connecting from the client to the Cluster WebUI and the MinIO service (in this article, http://drtest.example.com:9000/ ).

Failover from server1 to server2 in the N. Virginia region and verify that you can access the MinIO service without any issues.

Failover from server2 in the N. Virginia region to server3 in the Oregon region and verify that you can access the MinIO service without any issues. Additionally, by deleting the replication settings, the Amazon EFS file system on the Oregon region becomes writable, and you can verify that we can save files to the bucket in the MinIO service.

Failing back to the N. Virginia region without prior preparation results in the failure of the EXEC resource (exec-EFS) to start. This occurs because the replication setting has been deleted, leading to unsynchronized file systems between the N. Virginia and Oregon regions, causing the mount process within the EXEC resource script to be guarded. Therefore, before the failback process, it is necessary to configure replication on the file system, set the file system name for the file system replica, and add mount targets.

First, from the AWS Management Console in the N. Virginia region, either rename (modify the Name tag) or delete the initially created file system with the same name. Leaving it as is can result in multiple file systems with the same name, leading to malfunction.

Next, switch to the Oregon Region, select the file system (DRTest-EFS1), and click the "View details" button.

Click on the "Replication" tab located at the bottom of the file system details screen. Then, click the "Create replication" button.

In the "Create replication" screen, enter the following information. Other fields are optional.

Click the "Create replication" button. Return to the file system details screen of DRTest-EFS1, check "Replication" at the bottom of the screen, and click the link for the destination file system's file system ID (fs-xxxxxxxxxxxxxxxxx).

Switching to the N. Virginia region displays the detail screen of the destination file system (the replicated file system). Click the "Tags" tab at the bottom of the screen.

Add a Name tag with the same name as the file system name in the Oregon region.

Next, click the "Network" tab.
Click the "Create mount target" button.

Enter the following items.

Click the "Save" button.

Failback from server3 in the Oregon region to server1 in the N. Virginia region and verify that you can access the MinIO service and that previously saved files exist.

Note that replication settings are similarly removed for failback across regions. Therefore, the following procedures will be executed.

This time, we tried constructing a three-node HA cluster configuration in a multi-region and multi-AZ environment using Amazon EFS with EXPRESSCLUSTER X 5.2. By leveraging cross-region replication of Amazon EFS, we were able to confirm that data could be shared between regions and that cross-regional failover was possible.

The replication settings and creation of mount targets described in "4.3 Preparing for Failback" can also be performed using scripts of the EXEC resource. However, manual operations were chosen this time to simplify the failover process. Consider scripting them if necessary.

If you consider introducing the configuration described in this article, you can perform a validation with the trial module of EXPRESSCLUSTER. Please do not hesitate to contact us if you have any questions.