Active/Passive Cluster With Pacemaker, Corosync and DRBD on CentOS 7: Part 1 – Cluster Installation

The following is part 1 of a 4 part series that will go over an installation and configuration of Pacemaker, Corosync, Apache, DRBD and a VMware STONITH agent.

The aim here is to build an active/passive Pacemaker cluster with Apache and DRBD.

Before We Begin

Pacemaker is a sophisticated, feature-rich, and widely deployed cluster resource manager for the Linux platform. At its core, Pacemaker is a distributed finite state machine capable of co-ordinating the startup and recovery of inter-related services across a set of machines.

Pacemaker achieves maximum availability for cluster services (aka resources) by detecting and recovering from node and resource-level failures by making use of the messaging and membership capabilities provided by a preferred cluster infrastructure (either OpenAIS or Heartbeat).

Pacemaker is a continuation of the CRM (aka v2 resource manager) that was originally developed for Heartbeat but has since become its own project.

We will build a failover cluster, meaning that services may be spread over all cluster nodes.

Pacemaker Stack

A Pacemaker stack is built on five core components:

1. libQB – core services (logging, IPC, etc),
2. Corosync – membership, messaging and quorum,
3. Resource agents – a collection of scripts that interact with the underlying services managed by the cluster,
4. Fencing agents – a colllection of scripts that interact with network power switches and SAN devices to isolate cluster members,
5. Pacemaker itself.

As of RHEL 6.5, pcs and pacemaker are fully supported, see here for more info: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/s1-pacemaker65-70-HAAR.html.

The pcs package provides a command-line tool for configuring and managing the corosync and pacemaker utilities.

RHEL 7 replaced RGManager with Pacemaker for managing cluster resources and recovering from node failures.

Pacemaker Configuration Tools

1. crmsh – the original configuration shell for Pacemaker,
2. pcs – Pacemaker/Corosync Configuration System, an alternate vision for a full cluster lifecycle configuration shell and web based GUI.

Note that the original cluster shell (crmsh) is no longer available on RHEL.

The pcs command line interface provides the ability to control and configure corosync and pacemaker.

Linux-HA Best Practise

For resilience, every cluster should have at least two Corosync (read: heartbeat) rings and two fencing devices, to eliminate a single point of failure.

Marking

The convention followed in the series is that [ALL] # denotes a command that needs to be run on all cluster machines.

Software

Software used in the series:

1. CentOS Linux release 7.2.1511 (Core)
2. pacemaker-1.1.13
3. corosync-2.3.4
4. pcs-0.9.143
5. resource-agents-3.9.5
6. fence-agents-vmware-soap-4.0.11
7. drbd-8.4

Networking, Firewall and SELinux Configuration

We will build a two-node active/passive cluster using Pacemaker and Corosync.

We have two CentOS 7 virtual machines on VMware, named vm-pcmk01 and vm-pcmk02.

Networking

The following networks will be in use:

1. 10.247.50.0/24 – LAN with access to the Internet,
2. 172.16.21.0/24 – non-routable cluster heartbeat vlan for Corosync,
3. 172.16.22.0/24 – non-routable cluster heartbeat vlan for DRBD.

Hostnames and IPs which we have allocated:

Hostname	LAN IP
pcmk01, vm-pcmk01	10.247.50.211
pcmk02, vm-pcmk02	10.247.50.212
pcmk-vip (floating cluster resource)	10.247.50.213
Hostname	Corosync IP
pcmk01-cr	172.16.21.11
pcmk02-cr	172.16.21.12
Hostname	DRBD IP
pcmk01-drbd	172.16.22.11
pcmk02-drbd	172.16.22.12

The /etc/hosts file entries look as follows:

[ALL]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.247.50.211 pcmk01 vm-pcmk01
10.247.50.212 pcmk02 vm-pcmk02
10.247.50.213 pcmk-vip
172.16.21.11 pcmk01-cr
172.16.21.12 pcmk02-cr
172.16.22.11 pcmk01-drbd
172.16.22.12 pcmk02-drbd

Network configuration for the first node can be seen below, it is the same for the second node except the IPs which are specified above.

[pcmk01]# cat /etc/sysconfig/network-scripts/ifcfg-ens192
#LAN
NAME="ens192"
DEVICE="ens192"
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
PEERDNS="yes"
IPV4_FAILURE_FATAL="yes"
IPV6INIT="no"
ONBOOT="yes"
IPADDR="10.247.50.211"
PREFIX="24"
GATEWAY="10.247.50.1"
DNS1="8.8.8.8"
DNS2="8.8.4.4"

[pcmk01]# cat /etc/sysconfig/network-scripts/ifcfg-ens224
#Corosync ring0
NAME="ens224"
DEVICE="ens224"
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="no"
PEERDNS="no"
IPV4_FAILURE_FATAL="yes"
IPV6INIT="no"
ONBOOT="yes"
IPADDR="172.16.21.11"
PREFIX="24"

[pcmk01]# cat /etc/sysconfig/network-scripts/ifcfg-ens256
#DRBD
NAME="ens256"
DEVICE="ens256"
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="no"
PEERDNS="no"
IPV4_FAILURE_FATAL="yes"
IPV6INIT="no"
ONBOOT="yes"
IPADDR="172.16.22.11"
PREFIX="24"

Iptables

This article uses Iptables firewall. Note that CentOS 7 utilises FirewallD as the default firewall management tool.

Having spent a great amount of time learning Iptables (read: IP masquerade NAT postrouting WTH?), we can safely say that we know where all the bodies are buried… We’re keen in learning FirewallD one day, though.

The choice is obviously yours, however, here’s how to replace FirewallD service with Iptables:

[ALL]# systemctl stop firewalld.service
[ALL]# systemctl mask firewalld.service
[ALL]# systemctl daemon-reload
[ALL]# yum install -y iptables-services
[ALL]# systemctl enable iptables.service
[ALL]# service iptables save

These are the iptables rules that we have in use:

[ALL]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -s 10.0.0.0/8 -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
-A INPUT -s 10.0.0.0/8 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 172.16.21.0/24 -d 172.16.21.0/24 -p udp -m multiport --dports 5405 -j ACCEPT
-A INPUT -s 172.16.21.0/24 -d 172.16.21.0/24 -p tcp -m multiport --dports 2224 -j ACCEPT
-A INPUT -s 10.0.0.0/8 -p tcp -m multiport --dports 2224 -j ACCEPT
-A INPUT -s 10.0.0.0/8 -p tcp -m multiport --dports 3121 -j ACCEPT
-A INPUT -s 10.0.0.0/8 -p tcp -m multiport --dports 21064 -j ACCEPT
-A INPUT -s 172.16.22.0/24 -d 172.16.22.0/24 -p tcp -m multiport --dports 7788,7789 -j ACCEPT
-A INPUT -p udp -m multiport --dports 137,138,139,445 -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -j LOG --log-prefix "iptables_input "
-A INPUT -j DROP

SELinux

SELinux is set to enforcing mode.

Install Pacemaker and Corosync

[ALL]# yum install -y pcs

The pcs will install pacemaker, corosync and resource-agents as dependencies.

For SELinux management:

[ALL]# yum install -y policycoreutils-python

Set a password for the hacluster user:

[ALL]# echo "passwd" | passwd hacluster --stdin

Start and enable the service:

[ALL]# systemctl start pcsd.service
[ALL]# systemctl enable pcsd.service

Configure Corosync

Authenticate as the hacluster user. Note that we use a dedicated Corosync interface for this.

[pcmk01]# pcs cluster auth pcmk01-cr pcmk02-cr -u hacluster -p passwd
pcmk01-cr: Authorized
pcmk02-cr: Authorized

Authorisation tokens are stored in the file /var/lib/pcsd/tokens.

Generate and synchronise the Corosync configuration

[pcmk01]# pcs cluster setup --name test_webcluster pcmk01-cr pcmk02-cr

Start the cluster on all nodes:

[pcmk01]# pcs cluster start --all

Optionally, depending on requirements, we can enable cluster services to start on boot:

[ALL]# pcs cluster enable --all

Verify Corosync installation:

[pcmk01]# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
        id      = 172.16.21.11
        status  = ring 0 active with no faults

[pcmk01]# pcs status corosync

Membership information
----------------------
    Nodeid      Votes Name
         1          1 pcmk01-cr (local)
         2          1 pcmk02-cr

Corosync configuration for future references:

[pcmk01]# cat /etc/corosync/corosync.conf
totem {
version: 2
secauth: off
cluster_name: test_webcluster
transport: udpu
}

nodelist {
  node {
        ring0_addr: pcmk01-cr
        nodeid: 1
       }
  node {
        ring0_addr: pcmk02-cr
        nodeid: 2
       }
}

quorum {
provider: corosync_votequorum
two_node: 1
}

logging {
to_syslog: yes
}

Let us check the cluster status now:

[pcmk01]# pcs status
Cluster name: test_webcluster
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Sat Dec  12 15:24:14 2015          Last change: Sat Dec 12 15:24:08 2015 by hacluster via crmd on pcmk02-cr
Stack: corosync
Current DC: pcmk02-cr (version 1.1.13-a14efad) - partition with quorum
2 nodes and 0 resources configured

Online: [ pcmk01-cr pcmk02-cr ]

Full list of resources:


PCSD Status:
  pcmk01-cr: Online
  pcmk02-cr: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

We can also see the raw (XML) cluster configuration and status by using the following commands:

[pcmk01]# pcs cluster cib
[pcmk01]# cibadmin -Q

If we inspect the raw output, we can see that the Pacemaker configuration XML file contains the following sections:

1. <configuration>
2. <nodes>
3. <resources>
4. <constraints>
5. <status>

Disable STONITH

Check the list of available STONITH agents, it should be empty:

[pcmk01]# pcs stonith list

Disable STONITH for now as we don’t have any agents installed, however, we will configure it later in the series:

[pcmk01]# pcs property set stonith-enabled=false
[pcmk01]# crm_verify -LV

In production environments it is vitally important to enable STONITH.

Be advised that the use of stonith-enabled=false is completely inappropriate for a production cluster and may cost you your precious job.

STONITH/fencing actions should be taken into account in all cluster setups, but especially when running a dual-primary setup, since in such case access to data is available from more then one node at the same time.

Disable Quorum

You may get the following error when using STONITH:

ERROR: "Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore)"

Check the quorum property:

[pcmk01]# pcs property list --all|grep quorum
 no-quorum-policy: stop

A cluster has quorum when more than half of the nodes are online. Pacemaker’s default behavior is to stop all resources if the cluster does not have quorum. However, this does not make much sense in a two-node cluster; the cluster will lose quorum if one node fails.

We can tell Pacemaker to ignore quorum by setting the no-quorum-policy:

[pcmk01]# pcs property set no-quorum-policy=ignore

We now have a pacemaker cluster installed and running on two nodes.

References

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Clusters_from_Scratch/index.html