Global Site
Displaying present location in the site.
May 11th, 2023
Machine translation is used partially for this article. See the Japanese version for the original article.
Introduction
We tried building an HA cluster for an application in Docker container. By starting, stopping, and monitoring Docker containers from EXPRESSCLUSTER, Docker applications can be easily made redundant with a small number of nodes to improve availability. Please refer to this article when HA clustering is required in cases where applications are provided only in Docker container format.
This article is divided into two parts: "Part 1", which performs everything from the advance preparation in the Windows environment to creation of Docker container, and "Part 2", which configures EXPRESSCLUSTER settings for Docker services and container control. This time, we will introduce "Part 2". For "Part 1", please refer to 
here.
- *Note
Because of the cause that is probably same as the issue below, we have confirmed the problem that the execution of the wsl command described in this article fails depending on the environment. Therefore, it may not be possible to execute the wsl command in the EXPRESSCLUSTER script resource.
Contents
- 1. HA Cluster Building Procedure (EXPRESSCLUSTER Setting)
- 1.1 Register an Account for Script Execution
- 1.2 Add a Script Resource (Docker Service)
- 1.3 Add a Script Resource (Virtual Disk)
- 1.4 Add a Script Resource (Docker Container for Database)
- 1.5 Add a Script Resource (Docker Container for Application)
- 1.6 Add a Custom Monitor Resource (Docker Service)
- 1.7 Add a Custom Monitor Resource (Virtual Disk)
- 1.8 Add a PostgreSQL Monitor Resource
- 1.9 Add a Custom Monitor Resource (Docker Container for Application)
- 2. Checking the Operation
1. HA Cluster Building Procedure (EXPRESSCLUSTER Setting)
Add some resources for mounting a virtual disk and starting/stopping Docker service and Docker Containers in the failover group of EXPRESSCLUSTER.
- Failover group (failover1)
- - Script resource (script-DockerService)
- - Script resource (script-VirtualDisk)
- - Script resource (script-DockerDB)
- - Script resource (script-DockerApp)
- Monitor resources
- - Custom monitor resource (genw-DockerService)
- - Custom monitor resource (genw-VirtualDisk)
- - PostgreSQL monitor resource(psqlw-DockerDB)
- - Custom monitor resource (genw-DockerApp)
- *The scripts introduced in this article are samples and are not guaranteed to work.
- *The sleep values in each script are for reference. It also simplifies error handling and does not perform an exit code check if the script resource fails to be deactivated. Please satisfactorily consider the actual use according to your requirement.
1.1 Register an Account for Script Execution
First, register the [Administrator] in the [Account] tab of [Cluster Properties]. It is required to set Administrator as a user to run a script to be added later.
A script resource is run as a system account (SYSTEM) by default. However, the wsl command does not work as a system account, so you must change the setting to run it as the Administrator user.
Start Cluster WebUI, and switch to [Config mode]. Click [Cluster Properties] icon to the right of the cluster name.
On the [Account] tab, press the [Add], and then register Administrator.
1.2 Add a Script Resource (Docker Service)
Add a script resource to start and stop the Docker service.
■Script resource (script-DockerService)- [Dependency] tab
- - Dependent Resources: md-Docker
- [Details] tab
- > Scripts
- - Start Script: start.bat
- - Stop Script: stop.bat
- - User Script: Activate_DockerService.ps1
- - User Script: Deactivate_DockerService.ps1
- > Tuning
- - (Start) Normal Return Value: 0
- - Exec User: Administrator
Register scripts.
The content of each script is as follows.
rem ***************************************
rem start.bat
rem ***************************************
if "%CLP_EVENT%" == "RECOVER" exit /b 0
cd %CLP_SCRIPT_PATH%
PowerShell ".\Activate_DockerService.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
rem start.bat
rem ***************************************
if "%CLP_EVENT%" == "RECOVER" exit /b 0
cd %CLP_SCRIPT_PATH%
PowerShell ".\Activate_DockerService.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
- stop.bat
rem ***************************************
rem stop.bat
rem ***************************************
cd %CLP_SCRIPT_PATH%
PowerShell ".\Deactivate_DockerService.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
rem stop.bat
rem ***************************************
cd %CLP_SCRIPT_PATH%
PowerShell ".\Deactivate_DockerService.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
- Activate_DockerService.ps1
wsl -e sudo service docker start
if (! $?) {
Write-Error "Failed to start docker service."
exit 2
}
sleep 5
$ip = wsl -e hostname -I | % {$_.Split(" ")[0]}
if (! $?) {
Write-Error "Failed to get ip address of WSL."
wsl -e sudo service docker stop
exit 3
}
netsh interface portproxy add v4tov4 listenaddress=* listenport=80 connectaddress=$ip connectport=80
if (! $?) {
Write-Error "Failed to set portproxy."
wsl -e sudo service docker stop
exit 4
}
exit 0
if (! $?) {
Write-Error "Failed to start docker service."
exit 2
}
sleep 5
$ip = wsl -e hostname -I | % {$_.Split(" ")[0]}
if (! $?) {
Write-Error "Failed to get ip address of WSL."
wsl -e sudo service docker stop
exit 3
}
netsh interface portproxy add v4tov4 listenaddress=* listenport=80 connectaddress=$ip connectport=80
if (! $?) {
Write-Error "Failed to set portproxy."
wsl -e sudo service docker stop
exit 4
}
exit 0
- Deactivate_DockerService.ps1
netsh interface portproxy delete v4tov4 listenport=80 listenaddress=*
if (! $?) {
Write-Error "Warning: Failed to unset portproxy."
}
wsl -e sudo service docker stop
if (! $?) {
Write-Error "Failed to stop docker service."
exit 2
}
exit 0
if (! $?) {
Write-Error "Warning: Failed to unset portproxy."
}
wsl -e sudo service docker stop
if (! $?) {
Write-Error "Failed to stop docker service."
exit 2
}
exit 0
Next, click [Tuning].
In the [Script Resource Tuning Properties] window, set the values as follows.
- - (Start) Normal Return Value: 0
- - Exec User: Administrator
1.3 Add a Script Resource (Virtual Disk)
Add a script resource to mount and unmount the virtual disk.
■Script resource (script-VirtualDisk)- [Dependency] tab
- - Dependent Resources: script-DockerService
- [Details] tab
- > Scripts
- - Start Script: start.bat
- - Stop Script: stop.bat
- - User Script: Config.ps1
- - User Script: Mount_VirtualDisk.ps1
- - User Script: Unmount_VirtualDisk.ps1
- > Tuning
- - (Start) Normal Return Value: 0
- - Exec User: Administrator
Register scripts.
The content of each script is as follows.
rem ***************************************
rem start.bat
rem ***************************************
if "%CLP_EVENT%" == "RECOVER" exit /b 0
cd %CLP_SCRIPT_PATH%
PowerShell ".\Mount_VirtualDisk.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
rem start.bat
rem ***************************************
if "%CLP_EVENT%" == "RECOVER" exit /b 0
cd %CLP_SCRIPT_PATH%
PowerShell ".\Mount_VirtualDisk.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
- stop.bat
rem ***************************************
rem stop.bat
rem ***************************************
cd %CLP_SCRIPT_PATH%
PowerShell ".\Unmount_VirtualDisk.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
rem stop.bat
rem ***************************************
cd %CLP_SCRIPT_PATH%
PowerShell ".\Unmount_VirtualDisk.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
- Config.ps1
$VHDX_FILE="M:\nextcloud.vhdx"
$DATA_UUID="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
$DATA_DIR="/mnt/data"
$DATA_UUID="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
$DATA_DIR="/mnt/data"
- *The above configuration is an example. Set the actual value obtained in "Part 1".
. ".\Config.ps1"
$retval=0
try {
$ret = Mount-DiskImage $VHDX_FILE -ErrorAction "Stop"
sleep 1
} catch {
Write-Error "Failed to mount vhdx file."
exit 2
}
try {
try {
wsl --mount $ret.DevicePath --bare
if (! $?) {
$retval=3
throw "Failed to attach block devices."
}
sleep 3
wsl -e sudo mount UUID=$DATA_UUID $DATA_DIR
if (! $?) {
$retval=4
throw "Failed to mount device in WSL."
}
} catch {
wsl --unmount
sleep 10
Dismount-DiskImage $VHDX_FILE > $null
throw $_
}
} catch {
Write-Error $_
exit $retval
}
exit 0
$retval=0
try {
$ret = Mount-DiskImage $VHDX_FILE -ErrorAction "Stop"
sleep 1
} catch {
Write-Error "Failed to mount vhdx file."
exit 2
}
try {
try {
wsl --mount $ret.DevicePath --bare
if (! $?) {
$retval=3
throw "Failed to attach block devices."
}
sleep 3
wsl -e sudo mount UUID=$DATA_UUID $DATA_DIR
if (! $?) {
$retval=4
throw "Failed to mount device in WSL."
}
} catch {
wsl --unmount
sleep 10
Dismount-DiskImage $VHDX_FILE > $null
throw $_
}
} catch {
Write-Error $_
exit $retval
}
exit 0
- Unmount_VirtualDisk.ps1
. ".\Config.ps1"
wsl -e sudo umount $DATA_DIR
if (! $?) {
Write-Error "Warning: Failed to unmount device. Force to detach it in the next step."
}
wsl --unmount
if (! $?) {
Write-Error "Failed to detach all devices."
exit 2
}
sleep 10
Dismount-DiskImage $VHDX_FILE -ErrorAction "Stop" > $null
if (! $?) {
Write-Error "Failed to dismount vhdx file."
exit 3
}
exit 0
wsl -e sudo umount $DATA_DIR
if (! $?) {
Write-Error "Warning: Failed to unmount device. Force to detach it in the next step."
}
wsl --unmount
if (! $?) {
Write-Error "Failed to detach all devices."
exit 2
}
sleep 10
Dismount-DiskImage $VHDX_FILE -ErrorAction "Stop" > $null
if (! $?) {
Write-Error "Failed to dismount vhdx file."
exit 3
}
exit 0
Next, click [Tuning].
In the [Script Resource Tuning Properties] window, set the values as follows.
- - (Start) Normal Return Value: 0
- - Exec User: Administrator
1.4 Add a Script Resource (Docker Container for Database)
Add a script resource to start and stop the Docker container for database (nextcloud-db).
■Script resource (script-DockerDB)- [Dependency] tab
- - Dependent Resources: script-VirtualDisk
- [Details] tab
- > Scripts
- - Start Script: start.bat
- - Stop Script: stop.bat
- - User Script: Activate_DockerDB.ps1
- - User Script: Deactivate_DockerDB.ps1
- > Tuning
- - (Start) Normal Return Value: 0
- - Exec User: Administrator
Register scripts.
The content of each script is as follows.
rem ***************************************
rem start.bat
rem ***************************************
if "%CLP_EVENT%" == "RECOVER" exit /b 0
cd %CLP_SCRIPT_PATH%
PowerShell ".\Activate_DockerDB.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
rem start.bat
rem ***************************************
if "%CLP_EVENT%" == "RECOVER" exit /b 0
cd %CLP_SCRIPT_PATH%
PowerShell ".\Activate_DockerDB.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
- stop.bat
rem ***************************************
rem stop.bat
rem ***************************************
cd %CLP_SCRIPT_PATH%
PowerShell ".\Deactivate_DockerDB.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
rem stop.bat
rem ***************************************
cd %CLP_SCRIPT_PATH%
PowerShell ".\Deactivate_DockerDB.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
- Activate_DockerDB.ps1
wsl -e docker start nextcloud-db
if (! $?) {
Write-Error "Failed to start a docker container 'nextcloud-db'."
exit 2
}
exit 0
if (! $?) {
Write-Error "Failed to start a docker container 'nextcloud-db'."
exit 2
}
exit 0
- Deactivate_DockerDB.ps1
wsl -e docker stop nextcloud-db
if (! $?) {
Write-Error "Failed to stop a docker container 'nextcloud-db'."
exit 2
}
exit 0
if (! $?) {
Write-Error "Failed to stop a docker container 'nextcloud-db'."
exit 2
}
exit 0
Next, click [Tuning].
In the [Script Resource Tuning Properties] window, set the values as follows.
- - (Start) Normal Return Value: 0
- - Exec User: Administrator
1.5 Add a Script Resource (Docker Container for Application)
Add a script resource to start and stop the Docker container for application (nextcloud).
■Script resource (script-DockerApp)- [Dependency] tab
- - Dependent Resources: script-DockerDB
- [Details] tab
- > Scripts
- - Start Script: start.bat
- - Stop Script: stop.bat
- - User Script: Activate_DockerApp.ps1
- - User Script: Deactivate_DockerApp.ps1
- > Tuning
- - (Start) Normal Return Value: 0
- - Exec User: Administrator
Register scripts.
The content of each script is as follows.
rem ***************************************
rem start.bat
rem ***************************************
if "%CLP_EVENT%" == "RECOVER" exit /b 0
cd %CLP_SCRIPT_PATH%
PowerShell ".\Activate_DockerApp.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
rem start.bat
rem ***************************************
if "%CLP_EVENT%" == "RECOVER" exit /b 0
cd %CLP_SCRIPT_PATH%
PowerShell ".\Activate_DockerApp.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
- stop.bat
rem ***************************************
rem stop.bat
rem ***************************************
cd %CLP_SCRIPT_PATH%
PowerShell ".\Deactivate_DockerApp.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
rem stop.bat
rem ***************************************
cd %CLP_SCRIPT_PATH%
PowerShell ".\Deactivate_DockerApp.ps1; exit $LASTEXITCODE"
set ret=%ERRORLEVEL%
echo ret: %ret%
exit /b %ret%
- Activate_DockerApp.ps1
wsl -e docker start nextcloud
if (! $?) {
Write-Error "Failed to start a docker container 'nextcloud'."
exit 2
}
exit 0
if (! $?) {
Write-Error "Failed to start a docker container 'nextcloud'."
exit 2
}
exit 0
- Deactivate_DockerApp.ps1
wsl -e docker stop nextcloud
if (! $?) {
Write-Error "Failed to stop a docker container 'nextcloud'."
exit 2
}
exit 0
if (! $?) {
Write-Error "Failed to stop a docker container 'nextcloud'."
exit 2
}
exit 0
Next, click [Tuning].
In the [Script Resource Tuning Properties] window, set the values as follows.
- - (Start) Normal Return Value: 0
- - Exec User: Administrator
1.6 Add a Custom Monitor Resource (Docker Service)
Add a custom monitor resource to monitor the Docker service.
■Custom monitor resource (genw-DockerService)- [Monitor(common)] tab
- - Monitor Timing: Active
- - Target Resource: script-DockerService
- [Monitor(special)] tab
- - File: genw.bat
- - Exec User: Administrator
- [Recovery Action] tab
- - Recovery Action: Custom settings
- - Recovery Target: script-DockerService
Register a script.
The content of the script is as follows.
rem ***************************************
rem genw.bat
rem ***************************************
echo START
PowerShell -Command "wsl -e sudo service docker status; if (! $?) { exit 1 }; exit 0"
set ret=%ERRORLEVEL%
echo ret: %ret%
echo EXIT
exit /b %ret%
rem genw.bat
rem ***************************************
echo START
PowerShell -Command "wsl -e sudo service docker status; if (! $?) { exit 1 }; exit 0"
set ret=%ERRORLEVEL%
echo ret: %ret%
echo EXIT
exit /b %ret%
Select [Administrator] in [Exec User] list box.
1.7 Add a Custom Monitor Resource (Virtual Disk)
Add a custom monitor resource to monitor the virtual disk.
■Custom monitor resource (genw-VirtualDisk)- [Monitor(common)] tab
- - Monitor Timing: Active
- - Target Resource: script-VirtualDisk
- [Monitor(special)] tab
- - File: genw.bat
- - Exec User: Administrator
- [Recovery Action] tab
- - Recovery Action: Custom settings
- - Recovery Target: script-VirtualDisk
Register a script.
The content of the script is as follows.
- *For the variable "DATA_DIR" in line 4, set the same value as that specified in Config.ps1 of "1.3 Add a Script Resource (Virtual Disk)"
rem ***************************************
rem genw.bat
rem ***************************************
set DATA_DIR=/mnt/data
echo START
PowerShell -Command "wsl -e sudo mountpoint %DATA_DIR%; if (! $?) { exit 1 }; exit 0"
set ret=%ERRORLEVEL%
echo ret: %ret%
echo EXIT
exit /b %ret%
rem genw.bat
rem ***************************************
set DATA_DIR=/mnt/data
echo START
PowerShell -Command "wsl -e sudo mountpoint %DATA_DIR%; if (! $?) { exit 1 }; exit 0"
set ret=%ERRORLEVEL%
echo ret: %ret%
echo EXIT
exit /b %ret%
Select [Administrator] in [Exec User] list box.
1.8 Add a PostgreSQL Monitor Resource
Add a PostgreSQL monitor resource to monitor the Docker container for database (nextcloud-db).
■PostgreSQL monitor resource (psqlw-DockerDB)- [Monitor(common)] tab
- - Monitor Timing: Active
- - Target Resource: script-DockerDB
- [Monitor(special)] tab
- - Monitor Level: Level 2 (monitoring by update/select)
- - Database Name: nextcloud
- - IP Address: 127.0.0.1
- - Port Number: 5432
- - User Name: nextcloud
- - Password: xxxxxxxx (Specify the same password as when creating the container for PostgreSQL)
- - Monitor Table Name: PSQLWATCH
- [Recovery Action] tab
- - Recovery Action: Executing failover to the recovery target
- - Recovery Target: script-DockerDB
Download the PostgreSQL package from "PostgreSQL Downloads" because PostgreSQL monitor resource requires libpq.dll (postgresql-15.1-1-windows-x64.exe at the time of writing). During installation, select only "Command Line Tools" and execute the installation.
Add the path of libpq.dll to the system variable (C:\Program Files\PostgreSQL\15\bin for a default installation).
1.9 Add a Custom Monitor Resource (Docker Container for Application)
Add a custom monitor resource to monitor the Docker container for application (nextcloud).
Access the URL http://localhost:80/login and if it succeeds, it is considered normal.
- [Monitor(common)] tab
- - Monitor Timing: Active
- - Target Resource: script-DockerApp
- [Monitor(special)] tab
- - File: genw.bat
- - Exec User: Administrator
- [Recovery Action] tab
- - Recovery Action: Custom settings
- - Recovery Target: script-DockerApp
Register a script.
The content of the script is as follows.
rem ***************************************
rem genw.bat
rem ***************************************
echo START
PowerShell -Command "try { $ret = Invoke-WebRequest -Uri http://127.0.0.1:80/login -UseBasicParsing -ErrorAction Stop; if ($ret.StatusCode -ne '200') { exit 1 } } catch { exit 99 }; exit 0"
set ret=%ERRORLEVEL%
echo ret: %ret%
echo EXIT
exit /b %ret%
rem genw.bat
rem ***************************************
echo START
PowerShell -Command "try { $ret = Invoke-WebRequest -Uri http://127.0.0.1:80/login -UseBasicParsing -ErrorAction Stop; if ($ret.StatusCode -ne '200') { exit 1 } } catch { exit 99 }; exit 0"
set ret=%ERRORLEVEL%
echo ret: %ret%
echo EXIT
exit /b %ret%
Select [Administrator] in [Exec User] list box.
2. Checking the Operation
Check that the client machine can access the service of the HA clustered Docker container using the floating IP address, and is available without any problems in failure events.
- 1.Start the failover group (failover1) on server1.
- 2.From the client machine, access the floating IP address (172.16.1.100) and check that Nextcloud is accessible.
- 3.Log in to Nextcloud, and save any file to the online storage.
- 4.Shutdown server1 from Cluster WebUI, then the failover group moves from server1 to server2.
- 5.From the client machine, access the floating IP address (172.16.1.100) and confirm that you can open the file saved in step 3.
By building HA clustering of Docker, we were able to confirm that if one server went down, it failed over to the other server and the inherited data was accessible.
Conclusion
This time, we tried building an HA cluster for an application in Docker container.
By building HA clustering of Docker, it is possible to improve fault tolerance and availability while taking advantage of Docker. On Windows, there are some points to note described in this article when using it with WSL, but please refer to this procedure to build an HA cluster.
If you consider introducing the configuration described in this article, you can perform a validation with thetrial module of EXPRESSCLUSTER. Please do not hesitate to contact us if you have any questions. - * For support for software other than EXPRESSCLUSTER, please contact the respective vendor.