Displaying present location in the site.

We Tried Building an HA Cluster Using Amazon Elastic File System (Linux)

EXPRESSCLUSTER Official Blog

March 31st, 2023

Machine translation is used partially for this article. See the Japanese version for the original article.

Introduction

We tried building an HA cluster using Amazon Elastic File System (hereinafter called "Amazon EFS") on Amazon Web Services (hereinafter called "AWS"). Amazon EFS is a file storage that can be used on Linux Amazon EC2 instances (hereinafter called "EC2 instances").
This time, we will build an HA cluster using Amazon EFS as shared storage (location of taking over the data) between servers.

Even if the NAS is accessed from multiple servers at the same time, the NAS can perform exclusive control itself, basically preventing data corruption due to simultaneous access.
Therefore, in EXPRESSCLUSTER, the basic configuration is to use the NAS in the mounted state at all times without controlling the mounting/unmounting of the NAS.

This time, we will introduce the setting method and setting items for the configuration that uses the NAS in the mounted state at all times.

Contents

1. What is Amazon Elastic File System?

Amazon EFS is a file storage that can be used on Linux EC2 instances and can be used when you need to share storage between multiple servers.
Amazon EFS automatically grows and shrinks storage capacity as needed, so there is no need to reserve capacity in advance. In addition, Amazon EFS itself can be used from multiple Availability Zones(hereinafter called "AZ") and has redundancy, making it easier to manage than using an NFS server built with EC2 instances.

In the previous article, we introduced how to use Amazon EBS's Multi-Attach feature (Single-AZ configuration only) as a method of sharing data in an HA cluster on AWS. However, the Amazon EFS method introduced this time can realize a shared storage configuration in Multi-AZ, which expands the range of HA cluster configurations on AWS.

  • * Amazon EFS operates on Linux EC2 instances only and does not supported on Windows EC2 instances.

2. HA Cluster Configuration

This time, we will build an HA cluster based on VIP control and use popupMinIO for the HA cluster target application.
Amazon EFS is always mounted on server1 and server2 that servers in the cluster.

HA cluster configuration

3. HA Cluster Building Procedure

3.1 Preparing for Building an HA Cluster

3.1.1 VPC Settings

Create a VPC in advance. The VPC configuration is as follows.
In addition, enable both "DNS resolution" and "DNS hostnames" settings for the VPC in which the HA cluster is configured. Note that if the settings are disabled, the connection using the DNS name to Amazon EFS, created in next section, will fail.

N. Virginia region (us-east-1)

  • VPC(VPC ID : vpc-1234abcd)
  • - CIDR : 10.0.0.0/16
  • - Subnets
  • Subnet-A1 (Subnet ID:sub-1111aaaa) : 10.0.10.0/24
  • Subnet-A2 (Subnet ID:sub-2222aaaa) : 10.0.110.0/24
  • Subnet-B1 (Subnet ID:sub-1111bbbb) : 10.0.20.0/24
  • Subnet-B2 (Subnet ID:sub-2222bbbb) : 10.0.120.0/24
  • - RouteTables
  • Main (Route table ID:rtb-00000001)
  •  ・10.0.0.0/16   -> local
  •  ・0.0.0.0/0     -> igw-1234abcd (Internet Gateway)
  •  ・172.16.0.1/32 -> eni-1234abcd (ENI ID of server1)
  • Route-A (Route table ID:rtb-0000000a)
  •  ・10.0.0.0/16   -> local
  •  ・0.0.0.0/0     -> nat-1234abcd (NAT gateway 1)
  •  ・172.16.0.1/32 -> eni-1234abcd (ENI ID of server1)
  • Route-B (Route table ID:rtb-0000000b)
  •  ・10.0.0.0/16   -> local
  •  ・0.0.0.0/0     -> nat-5678efgh (NAT gateway 2)
  •  ・172.16.0.1/32 -> eni-1234abcd (ENI ID of server1)

VPC

3.1.2 Amazon EFS Settings

Create an Amazon EFS. For instructions on how to create an Amazon EFS and mount it on an EC2 instance, refer to popupGetting started with Amazon Elastic File System.
The DNS name of the Amazon EFS you created will be used when mounting from the EC2 instances, so write it down. And set security group of the Amazon EFS to be accessible only from the servers for HA cluster.

Also, when using Amazon EFS by mounting it on an EC2 instance, it is recommended to use the Amazon EFS mount helper. Using the Amazon EFS mount helper simplifies mounting by using Amazon EFS recommended mount options. Please refer to the following for how to use Amazon EFS mount helper.

If you do not use the Amazon EFS mount helper, refer to the following.

3.1.3 Mounting Amazon EFS File System Automatically Settings

Add the settings to "/etc/fstab" so that the servers in the cluster automatically mount Amazon EFS at startup. This time, to use the EFS mount helper, specify "efs" for the file system and mount it.
Execute the following on each server for the HA cluster.

Create a mount point (e.g. /mnt/efs).

# mkdir /mnt/efs

Add settings to "/etc/fstab" to automatically mount Amazon EFS (e.g. fs-xxxxxxxx) to the mount point (e.g. /mnt/efs).

fs-xxxxxxxx:/ /mnt/efs efs _netdev,tls 0 0

For more information of mounting Amazon EFS file system automatically, refer to the following.

3.1.4 Setting Dependency Between Services

Automatic mounting by "/etc/fstab" may start EXPRESSCLUSTER service even if Amazon EFS mounting is not completed. In that case, the application may start without the referenced Amazon EFS mounted, and may fail to start. Therefore, it is necessary to confirm that Amazon EFS is mounted before starting the EXPRESSCLUSTER service.
Execute the following on each server for the HA cluster.

Add the Amazon EFS mount unit (e.g. mnt-efs.mount) to the dependencies of the EXPRESSCLUSTER service file (/usr/lib/systemd/system/clusterpro_evt.service).

[Unit]Description=EXPRESSCLUSTER event
Requires=mnt-efs.mount
After=network-online.target remote-fs.target mnt-efs.mount

(Omitted below)

  • * If Amazon EFS mount fails, EXPRESSCLUSTER service will not start.
  • * If "/mnt/efs" is specified as the mount point, the file name of the mount unit will be "mnt-efs.mount".
    Please refer to the following for naming rules for mount units.
[Reference]
popupsystemd.mount

3.1.5 MinIO Settings

Install MinIO on the servers for the HA cluster. Refer to the popupQuickstart for Linux for installation instructions. Please specify an arbitrary location for the installation destination of MinIO. This time, install to "/work" created under the root directory. After installation, check the operation of MinIO on each server. Start the MinIO service so that it refers to the mounted Amazon EFS.

# /work/minio server /mnt/efs --console-address :9090

After checking the operation of MinIO, stop the MinIO service on each server.

# pgrep minio | xargs kill

3.2 Building an HA Cluster Based on VIP Control

In this verification, EXPRESSCLUSTER X 5.0 (Internal Ver. Linux : 5.0.2-1).
As mentioned above, when using NAS with EXPRESSCLUSTER, we will introduce a configuration is to use the NAS in the mounted state at all times without controlling the mounting/unmounting of the NAS.

The configuration of EXPRESSCLUSTER is as follows.

  • EXPRESSCLUSTER
  • - Failover group (failover)
  • AWS Virtual IP resource(awsvip)
  • - IP address : 172.16.0.1
  • EXEC resource(exec_minio)
  • - Start script : Script to start MinIO
  • - Stop script : Script to stop MinIO
  • - Tuning -> Parameter -> Start Script : Asynchronous
  •  
  • - Monitors
  • Disk monitor resource
  • - Monitor Timing : Always
  • - Method : WRITE(FILE)
  • - Monitor Target : /mnt/efs/monitor.txt
  • - I/O size : 1byte
  • PID monitor resource
  • - Monitor Timing : Active
  • - Target Resource : exec_minio

3.2.1 Registering Group Resources

In EXPRESSCLUSTER's Failover Group, we register two group resources: AWS Virtual IP resource and EXEC resource.
The AWS Virtual IP resource realizes switching of the connection destination by rewriting the route table. Set "172.16.0.1" for the virtual IP address. Please refer to the following for detailed configuration instructions.

[Reference]
popupDocumentation - Setup Guides
  • Linux > Cloud > Amazon Web Services > EXPRESSCLUSTER X 5.0 for Linux HA Cluster Configuration Guide for Amazon Web Services

The EXEC resource controls the start and stop of HA cluster target applications. Scripts are used to start and stop applications. Write the scripts to start/stop MinIO in the start script(start.sh) and stop script(stop.sh) of the EXEC resource(e.g. exec_minio).

Start Script(start.sh)

/work/minio server /mnt/efs --console-address :9090

Stop Script(stop.sh)

pgrep minio
if [ $? -eq 0 ];
then
    pgrep minio | xargs kill
    exit $?
else
    exit 0
fi

Press the [Tuning] button to change the start script from "Synchronous" to "Asynchronous" mode.

Tuning for exec resource

3.2.2 Registering Monitor Resources

We register two monitor resources: Disk monitor resource and PID monitor resource.
The Disk monitor resource monitors whether Amazon EFS can be accessed. Select "Always" for [Monitor Timing].

Select "WRITE(FILE)" for [Method]. Specify any file in the mount point (e.g. /mnt/efs/monitor.txt) as [Monitor Target], and set "1 byte" for [I/O size].

Next, monitor the MinIO service using the PID monitor resource. For [Target Resource], specify the EXEC resource (e.g. exec_minio) that controls MinIO.

4. Checking the Operation

From the Cluster WebUI, check that the failover group has started normally on server1.

Connect to MinIO using the virtual IP address (172.16.0.1) and create a bucket (e.g. test).

Upload a text file (e.g. test.txt) to the bucket.

To check the failover operation, manually move the failover group to server2.

After checking that the failover group has started on server2, connect to MinIO again using the virtual IP address (172.16.0.1). It is successful if you can refer to the text file created before manual failover.

Conclusion

This time, we tried building an HA cluster using Amazon EFS.
We were able to confirm that it works without any problems even without controlling Amazon EFS mounting/unmounting from EXPRESSCLUSTER.
When building a cluster using NAS with EXPRESSCLUSTER, please refer to this procedure to build an HA cluster.

If you consider introducing the configuration described in this article, you can perform a validation with the popuptrial module of EXPRESSCLUSTER. Please do not hesitate to contact us if you have any questions.