HAProxy implementation Case Study
This tutorial covers HAProxy Deployment on Firewall and
SELinux enabled Centos7 systems.
First of all lets get an overall idea about my situation.
I’ve bought a domain call mycompany.com so all of my hosted sites should be followed by this main domain.
As an example, if someone look for london.mycompany.com he should
reach to London server, if someone look for chicago.mycompany.com he should
reach to Chicago server
So, I’ve created a Cloudflare account and point
mycompany.com to our public IP address and created two CNAME entries from Cloudflare by adding London and Chicago.
From the Cloudflare all the request to mycompany.com will
forward to our public address and HAProxy may read those requests
and Process them and forward them accordingly between two IIS servers.
Note that all HTTPs connection should terminated at HAProxy. Please see my post
HTTPS for HAProxy
In this chapter we will look in to Load balancing and ACL
based traffic routing between IIS web servers
As the first step I will add hosts entries for my hosts file
vi /etc/hosts
10.0.3.121 iiswebsrv01
10.0.3.131 iiswebsrv02
You may ping and verify the connectivity between HAProxy
server and your web servers.
Download and Enable the EPEL Repositories and install the
HAProxy
wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -ivh epel-release-latest-7.noarch.rpm
yum -y install HAProxy
After Successful installation of HAProxy, you can start configuration
on HAProxy.
Prior to edit the configuration file its recommended to keep
a backup of existing HAProxy configuration file
cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg_bak
Now Let’s look at our HAProxy configurations
HAProxy configuration file contains four types sections
- Global Section which contains global configurations
- Default Section for default configurations
- Frontend sections which expose to Public to accepts the requests
- Backend sections which define who to be should serve the requests which are forwarding from front end section.
global
log 127.0.0.1 local2 #1
maxconn 1024 #2
user haproxy #3
group haproxy #4
daemon #5
stats socket /var/run/haproxy.sock mode 600 level admin #6
tune.ssl.default-dh-param 2048 #7
1. In global section it is configured where to store our HAProxy logs, in here our HAProxy logs will be stored by using local rsyslog server
2. Maximum Connection count which is supposed accommodate
using my HAProxy Server
3. Define HAProxy user which is already created with the
installation
5. State that HAProxy should run as a daemon process
6. In HAProxy you can monitor the performance by using status page or socket, this for socket usage.
7. Define the Maximum size of the temporary DHE Key for TLS
Now we are done with our global configurations…
It’s time to start our configurations on HAProxy Default
Configuration
In default section we configure the default parameter for
all frontend and backend sections
defaults
log global #1
option tcplog #2
option dontlognull #3
retries 3 #4
option redispatch #5
maxconn 1024 #6
timeout connect 50000ms #7
timeout client 500000ms #8
timeout server 500000ms #9
1. Define the log mode to use which we have already dud in Global configuration
2. option tcplog will logging will keep the logs on tcp (Layer 4) level.
Note that you may also use httplog if you’re willing to log on http level
3. dontlognull will filter the log entries, this will keep your logs clean (By default even a simple port probe will produce a log, due to that HAProxy in order to avoid these type of log entries you may use this option)
4. This will set the number of retries after a fail attempts to the server
5. If a server designated by a cookie and it is down, clients may stick to this because they cannot flush the cookie, so this client will not be able be to the access the particular service anymore. So, this will allow the HAProxy to break their persistence and redistribute them to a working server.
6. Define the maximum connection count
7. Set the maximum time to a client to wait for the response
8. Set maximum inactive time on the client side
9. Set maximum inactive time on the server side
Now our basic Configuration is done let’s start to
configuring the parameters for frontends and backends.
In frontend sections it will accept the traffic from public
entities and process them, and it will forward the traffic to relevant backend.
In following front end section, I will demonstrate how to read HTTP headers and process them with basic ACLs and forward them to relevant
backend sections.
frontend http_handler
bind *:80 #1
bind *:443 ssl crt /etc/haproxy/certs/hrms.itcstaging.com-0001.pem #2
mode http #3
option httplog #4
log global #5
acl acl_london hdr_beg(host) -i london #6
acl acl_chicago hdr_beg(host) -i chicago #7
use_backend be_london if acl_london #8
use_backend be_chicago if acl_chicago #9
default_backend be_welcome #10
This frontend will accept all http and https request and
process them with a ACL.
1,2 In order to accept the traffic from port 80 (http) and
443 (https), shall we bind both 80,443 ports to a front end. Note that for https traffic it is required to have a SSL certificate.
Here in this tutorial I’ve created a SSL certificate using let’s
encrypt (https://letsencrypt.org) (
Please see my post HTTPs for HAProxy)
3 since our primary objective is to read those HTTP/HTTPs headers
and forward the traffic accordingly we may use ‘mode http’, if you use the TCP
mode instead you load balancing will be based on layer 4, if so we won’t be
able to read the HTTP header in our ACL
4 Here I have enable HTTP logging of HTTP/HTTPs Request
5 Add logs to the global syslog service
In HAProxy an ACL can define the “acl” keyword,
ACLs can be defined in either backend or fronted. (In our scenario ACL have to
be defined in the frontend section)
acl
<Name of the ACL> <criterion> [flags][option] values
6,7 I’ve initiated two acls for my two sites one
london(acl_london) and one for chicago(acl_chicago), now as per the requirement
it’s required to read http headers and forward the traffic based on the
content of the http headers.
In here I’ve used hdr_beg criterion which will test the http
header with defined keyword. alternately you may use hdr(host)
which will test against the exact
match of the header content.
Here with hdr_beg(host) I’ve use -i option ignore the case.
So, if there is any http request which contain london key
word will activate the ACL “acl_london” and if there is any http
request whoch contain chicago it will activate “acl_chicago”
Now we have defined the conditions for the ACLs, now we
should instruct HAProxy to what should happen if that particular ACL entry is
true
8,9 use_backend can be used to switch backends depends on
the ACL output
use_backend
<desired backend> {if | unless} [Condition]
In our scenario if acl_london is true it should use the
london backend (be_london), if acl_chicago is true it should use the chicago
backend (be_chicago)
10. Now in practical scenarios, some HTTP requests may hit
our frontends but failed on the ACLs, which means our ACLs happens to unidentified
the which backend to use, on those kinds of scenarios we may use
default_backend option which is not compulsory here
Now traffic come to our frontends and test against the ACLs,
it’s time to define our backends to accommodate HTTP request and forward
them to desired destination
backend be_london
balance leastcon #1
redirect scheme https if !{ ssl_fc } #2
option forceclose #3
option forwardfor #4
stick match src #5
stick-table type ip size 200k expire 30m #6
mode http #7
reqadd X-Forwarded-Proto: http #8
cookie SERVERID insert indirect nocache #9
option httpchk GET /check.aspx?testsrv=londonsrv01:8080 #10
http-check expect string 200 OK #11
server london_01 iiswebsrv01:94 cookie L01 check #12
server london_02 iiswebsrv02:94 cookie L02 check #12
1 balance key word state the balancing which balancing algorithm
to use when HAProxy selecting the destination servers
balance
<algorithms> [argument]
HAProxy proxy provides various balancing algorithms to balance
the load between the servers
- leastconcon : Selects the server with the least number of connections–This is recommended for longer sessions. Servers in the same backend are also rotated in a round-robin fashion.
- roundrobin : Round Robin selects servers in turns. This is the default algorithm.
- source : This selects which server to use based on a hash of the source IP i.e. user’s IP address. This is one method to ensure that a user will connect to the same server.
2 Now for the security purposes I prefer only to use HTTPS. so in the backends, here we forcefully filter the traffic only to use HTTPs.
Note that In TCP mode, HAProxy doesn’t actually even
terminate SSL, it just passes the packets on to the backend.
For additional information, ssl_fc: Returns true when
the front connection was made via an SSL/TLS transport layer and is locally
deciphered.
4 This will enable X-Forwarding the HTTP connections.
What is
X-Forwarding, HAProxy act as a revers proxy which we already know. when it comes to your server side, the server will only see the ip address of HAProxy
server as the client address which sometimes it may cause some issues. By using
X-Forwarding, HAProxy will append the original IP address of the client when
requests are sending to the server.
3 By default HAProxy operates in keep alive mode with the connection
and keep connection idle from both the Client and Server.
Simply ” option
forceclose” will check if “Connection: close” header is already set
in each direction and will add the Connection close if not including headers
and will kill the connection. (Also, any “Connection” header
different from “close” will also be removed)
5 stick match statement define a rule to extract the
stickiness criterion from an incoming request or connection.
6 This is to configure the sticky table. These stick
tables will keep learned data on the memory from the connection. Note that restarting the service will remove those sticky entries (To overcome this issue
you may implement HAProxy peering with secondary HAProxy server).
7 mode http is optional since we have already define it
on default Section .
8 In here we are adding a header to all HTTP request’s which
are passing through this backend (Header transformations only apply to traffic
which passes through HAProxy)
9 Adding Cookie values to HTTP requests. This value will
be checked in incoming requests, and the first operational server possessing
the same value will be selected. In return, in cookie insertion or rewrite
modes, this value will be assigned to the cookie sent to the client.
10 Before forward the traffic to the destination servers it
is wise to check the availability of those servers using httpchk. It will point to “from where” HAProxy should get this information.
Additionally,
I’ve created a separate sites on both of my chicago and london servers which
will give us an output saying “200 OK”, based on the readiness of this
content HAProxy will figure it out that particular destination server is ready
serve. (I’ve put this sample html script later on this tutorial)
11 Now HAProxy test previous readings from the httpchk
and test it
It’s time to us to define our backend servers. Back end
servers can define using “server”.
server
<name> <server address>[:[port]] [param*]
12 Here in our configuration I’ve defined a name for
particular cookies from london_01 server.
The check
parameter will do the test based on our previous html script
And you may add the other Back End servers as well.
Now we are done with the configuration..
Once you complete the HAProxy configuration you may check the configurations by
haproxy -f /etc/haproxy/haproxy.cfg -c
To restart the HAProxy
systemctl restart haproxy
You may fall in to some SELinux related issue, which some destination ports are not
allowed by SELinux in Such a scenario you may use following command,
semanage port --add --type http_port_t --proto tcp <port>
You may require to open firewall ports, to do that use
firewall-cmd --permanent --add-port=tcp
firewall-cmd --reload
Complete Configuration Sample
global
log 127.0.0.1 local2
maxconn 1024
user haproxy
group haproxy
daemon
stats socket /var/run/haproxy.sock mode 600 level admin
tune.ssl.default-dh-param 2048
defaults
log global
option tcplog
option dontlognull
retries 3
option redispatch
maxconn 1024
timeout connect 50000ms
timeout client 500000ms
timeout server 500000ms
###########################
#Front Ends Configurations#
###########################
frontend http_handler
bind *:80
bind *:443 ssl crt /etc/haproxy/certs/hrms.itcstaging.com-0001.pem
mode http
option httplog
log global
acl acl_london hdr_beg(host) -i london
acl acl_chicago hdr_beg(host) -i chicago
use_backend be_london if acl_london
use_backend be_chicago if acl_chicago
default_backend be_welcome
###########################
#Back Ends Configurations#
###########################
backend be_london
balance leastcon
redirect scheme https if !{ ssl_fc }
option forceclose
option forwardfor
stick match src
stick-table type ip size 200k expire 30m
mode http
reqadd X-Forwarded-Proto: http
cookie SERVERID insert indirect nocache
option httpchk GET /check.aspx?testsrv=iiswebsrv01:8080
http-check expect string 200 OK
server london_01 iiswebsrv01:94 cookie L01 check
server london_02 iiswebsrv02:94 cookie L02 check
backend be_chicago
balance leastcon
redirect scheme https if !{ ssl_fc }
option forceclose
option forwardfor
stick match src
stick-table type ip size 200k expire 30m
mode http
reqadd X-Forwarded-Proto: http
cookie SERVERID insert indirect nocache
option httpchk GET /check.aspx?testsrv=iiswebsrv01:8080
http-check expect string 200 OK
server london_01 iiswebsrv01:94 cookie L01 check
server london_02 iiswebsrv02:94 cookie L02 check
Sample HTTP check html Script
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>
Mediatools Check
</title>
</head>
<body id="bodyID">
200 OK
</body>
</html>
