Setting up a Load-Balancing HAProxy Cluster with Keepalived

We will be setting up a load balancer using two main technologies to monitor cluster members and cluster services: Keepalived and HAProxy. Keepalived uses LVS to perform load balancing and failover tasks on active and passive LVS routers, while HAProxy performs load balancing and high-availability services to TCP and HTTP applications.

Software

Software used in this article:

1. CentOS 6
2. HAProxy 1.5.4
3. Keepalived 1.2.13

Networking and IP Addresses

Our network is set up as follows:

1. 10.8.8.0/24 – LAN with access to the Internet.

Hostnames and roles of the virtual machines we are going to use:

1. lvs-hap01 – the active HAProxy router with keepalived,
2. lvs-hap02 – the backup HAProxy router with keepalived,
3. lvs-hap03/lvs-hap04 – real servers, both running a pre-configured Apache webserver with SSL.

See the schema below for more information.

Two HAProxy load balancers are deployed as a failover cluster to protect the load balancer against outages.

Multi-Port Services and Firewall Marks

We are going to use multi-port services (HTTP and HTTPS), therefore firewall marks to bundle together different, but related protocols, are required.

Assigning firewall marks on both routers, the lvs-hap01 and the lvs-hap02:

# iptables -t mangle -A PREROUTING -p tcp -d 10.8.8.40/32 -m multiport --dport 80,443 -j MARK --set-mark 80

Where 10.8.8.40 is our virtual IP address. Save iptables rules so that they get restored after a reboot:

# service iptables save

HAProxy with Direct Routing

We are going to user direct routing with HAProxy.

Building a load balancer setup that uses direct routing provides increased performance benefits as direct routing allows the real servers to process and route packets directly to a requesting user rather than passing all outgoing packets through the LVS router. Direct routing reduces the possibility of network performance issues by relegating the job of the LVS router to processing incoming packets only.

On the lvs-hap01 router, allow Virtual Redundancy Routing Protocol (VRRP) traffic and save firewall rules:

# iptables -I INPUT -p vrrp -m comment --comment "VRRP" -j ACCEPT 
# service iptables save

In order for the Keepalived service to forward network packets properly to the real servers, each router node must have IP forwarding turned on in the kernel:

# sed -i 's/net.ipv4.ip_forward = 0/net.ipv4.ip_forward = 1/' /etc/sysctl.conf
# sysctl -p

Load balancing in HAProxy also requires the ability to bind to an IP address that is nonlocal, meaning that it is not assigned to a device on the local system. This allows a running load balancer instance to bind to a an IP that is not local for failover.

# echo "net.ipv4.ip_nonlocal_bind=1" >>/etc/sysctl.conf
# sysctl -p

Copy firewall and sysctl configuration to the backup lvs-hap02 router:

# scp /etc/sysconfig/iptables root@lvs-hap02:/etc/sysconfig/
# scp /etc/sysctl.conf root@lvs-hap02:/etc/

HAProxy Setup

HAProxy performs load-balancing management on layer 7 (application layer). It is a common use case to deploy an HAProxy for HTTP-based load balancing.

Installation

Install packages on both router nodes, the lvs-hap01 and the lvs-hap02:

# yum install -y haproxy

Enable on boot:

# chkconfig haproxy on

Configuration

We are going to configure HAProxy on the master router node, the lvs-hap01.

Backup the configuration file before editing:

# cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.org

Our HAProxy /etc/haproxy/haproxy.cfg configuration is below.

The 10.8.8.40.pem file contains our self-signed SSL certificate together with the private key.

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4096
    user        haproxy
    group       haproxy
    daemon

    ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

#---------------------------------------------------------------------
# Common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block.
#---------------------------------------------------------------------
defaults
    #mode                   tcp|http
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option 		    http-server-close

    # Since HAProxy works in reverse-proxy mode, the servers see its IP address as
    # their client address. This is sometimes annoying when the client's IP address
    # is expected in server logs. To solve this problem, the well-known HTTP header
    # "X-Forwarded-For" may be added by HAProxy to all requests sent to the server.
    # This header contains a value representing the client's IP address.
    option forwardfor       except 127.0.0.0/8

    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 2048

#---------------------------------------------------------------------
# Main frontend which proxys to the backends.
#---------------------------------------------------------------------
frontend lb-http
    bind 10.8.8.40:80
    reqadd X-Forwarded-Proto:\ http
    default_backend http-nodes

frontend lb-https
   bind 10.8.8.40:443 ssl crt /etc/ssl/certs/10.8.8.40.pem
   reqadd X-Forwarded-Proto:\ https
   default_backend http-nodes

#---------------------------------------------------------------------
# Round robin balancing between the various backends.
#---------------------------------------------------------------------
backend http-nodes
    balance roundrobin
    server  lvs-hap03 10.8.8.43:80 check
    server  lvs-hap04 10.8.8.44:80 check

#---------------------------------------------------------------------
# Stats.
#---------------------------------------------------------------------
listen stats 10.8.8.40:8080
    mode http
    stats enable
    stats uri /
    #stats hide-version
    stats realm HAProxy\ Statistics
    stats auth admin:passwd

A few of commonly used HAProxy load balancing algorithms are as follows:

roundrobin: round robin selects servers in turns. This is the default algorithm.

leastconn: selects the server with the least number of connections, it is recommended for longer sessions. Servers in the same backend are also rotated in a round-robin fashion.

source: this selects which server to use based on a hash of the source IP i.e. your user’s IP address. This is one method to ensure that a user will connect to the same server.

Start the HAProxy sevice:

# service haproxy start

Copy configuration to the backup lvs-hap02 router:

# scp /etc/haproxy/haproxy.cfg root@lvs-hap02:/etc/haproxy/

Do not forget to start the HAProxy service on the backup node.

Keepalived Setup

The keepalived daemon runs on both the active and the passive LVS routers. Both routers running keepalived use the Virtual Redundancy Routing Protocol (VRRP). The active router sends VRRP advertisements at periodic intervals, and if the backup routers fail to receive these advertisements, a new active router is elected.

Keepalived performs failover on layer 4 (transport layer), upon which TCP conducts connection-based data transmissions. When a real server fails to reply to simple timeout TCP connection, keepalived detects that the server has failed and removes it from the server pool.

Installation

Install packages on both router nodes, the lvs-hap01 and the lvs-hap02:

# yum install -y keepalived

Enable on boot:

# chkconfig keepalived on

Configuration

Backup the config file before editing.

# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.org

Configuration for the master lvs-hap01 node:

global_defs {
    notification_email {
        root@localhost
    }
    #notification_email_from keepalived@localhost
    #smtp_server 127.0.0.1
    #smtp_connect_timeout 30
}

# Script used to check if HAProxy is running
vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance LVS_HAP {
    #state MASTER|BACKUP
    state MASTER
    interface eth0
    virtual_router_id 51

    # 5 on master, 4 on backup
    # Note that priority is set in the VRRP router and defines the master
    priority 5
    advert_int 1
    authentication {
        #auth_type PASS|AH
        auth_type PASS
        auth_pass changeme
    }
    virtual_ipaddress {
        10.8.8.40
    }
    track_script {
        check_haproxy
    }
}

Configuration for the backup lvs-hap02 node:

global_defs {
    notification_email {
        #root@localhost
    }
    #notification_email_from keepalived@localhost
    #smtp_server 127.0.0.1
    #smtp_connect_timeout 30
}

# Script used to check if HAProxy is running
vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance LVS_HAP {
    state SLAVE
    interface eth0
    virtual_router_id 51

    # 5 on master, 4 on backup
    priority 4
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass changeme
    }
    virtual_ipaddress {
        10.8.8.40
    }
    track_script {
        check_haproxy
    }
}

Start the Keeplived service:

# /etc/init.d/keepalived start

At this point we should see the virtual IP address 10.8.8.40 attached to the eth0 interface:

# ip ad sh eth0
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:ff:41:00 brd ff:ff:ff:ff:ff:ff
    inet 10.8.8.41/24 brd 10.10.1.255 scope global eth0
    inet 10.8.8.40/32 scope global eth0
    inet6 fe80::a00:27ff:feff:4100/64 scope link 
       valid_lft forever preferred_lft forever

Real Servers

On each real server node (lvs-hap03 and lvs-hap04), run the following command for the VIP 10.8.8.40 and protocol combination intended to be serviced for the real server:

# iptables -t nat -A PREROUTING -p tcp -d 10.8.8.40 -m multiport --dport 80,443 -j REDIRECT

The command above will cause the real servers to process packets destined for the VIP and port that they are given. Make sure that firewall changes are saved and restored after a restart:

# service iptables save

References

http://www.keepalived.org/pdf/UserGuide.pdf
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Load_Balancer_Administration/index.html