Asymmetric routing, multiple default gateways on Linux with HAProxy

Why we may need multiple default gateways?

Nowadays, Application Delivery controllers (aka Load-Balancers) become the entry point for all the applications hosted in a company or administration.
That said, many different type of population could access the applications:
  * internal users from the LAN
  * partners through MPLS or VPNs
  * external users from internet

On the other side, applications could be hosted on different VLANs in the architecture:
  * internal LAN
  * external DMZ

The diagram below shows the “big picture” of this type of architecture:
multiple_default_gateways

Routing in the Linux network stack

I’m not going to deeply explain how it works, sorry… It would deserve a complete blog post :)
That said, any device connected on an IP network needs an IP address to be able to talk to other devices in its LAN. It also needs a default gateway to be able to reach devices which are located outside its LAN.
A Linux kernel can use a single default gateway at a time, but thanks to the metric you can configure many default gateways.
When needed, the Linux Kernel will parse the default gateway table and will use the one with the lowest metric. By default, when no metric is configured, the kernel attributes a metric 0.
Each metric must be unique in your Kernel IP stack.

How HAProxy can help in such situation??


Users access applications through a HAProxy bind. The bind can be hosted on any IP address available or not (play with your sysctl for this purpose) on the server.
By default, the traffic comes in HAProxy through this bind and HAProxy let the kernel choose the most appropriate default gateway to forward the answer to the client. As we’ve seen above, the most appropriate default gateway from the kernel point of view is the one with the lowest metric usually 0.

That said, HAProxy is smart enough to tell the kernel which network interface to use to forward the response to the client. Just add the statement interface ethX (where X is the id of the interface you want to use) on HAProxy bind line.
With this parameter, HAProxy can force the kernel to use the default gateway associated to the network interface ethX if it exists, otherwise, the interface with the lowest metric will be used.

Security concern


From a security point of view, some security manager would say that it is absolutely unsecure to plug a device in multiple DMZ or VLANs. They are right. But usually, this type of company’s business is very important and they can affoard one load-balancer per DMZ or LAN.
That said, there is no security breach with the setup introduced here. HAProxy is a reverse-proxy and so you don’t need to allow ip_forward between all interfaces for this solution to work.
I mean that nobody could use the loadbalancer as a default gateway to reach an other subnet bypassing the firewall…
Then only traffic allowed to pass through is the one load-balanced!

Configuration

The configuration below applies to the ALOHA Loadbalancer. Just update the content to match your Linux distribution configuration syntax.
The configuration is also related to the diagram above.

Network configuration


In your ALOHA, go in the Services tab, then edit the Network configuration.
To keep it simple, I’m not going to add any VRRP configuration.

service network eth0
    ########## eth0.
    auto on
    ip   address 10.0.0.2/24
    ip   route   default 10.0.0.1

service network eth1
    ########## eth1.
    auto on
    ip   address 10.0.1.2/24
    ip   route   default 10.0.1.1 metric 1

service network eth2
    ########## eth2.
    auto on
    ip   address 10.0.2.2/24
    ip   route   default 10.0.2.1 metric 2

service network eth3
    ########## eth3.
    auto on
    ip   address 10.0.3.2/24
    ip   route   default 10.0.3.1 metric 3

service network eth4
    ########## eth4.
    auto on
    ip   address 10.0.4.2/24
    ip   route   default 10.0.4.1 metric 4

The routing table from the ALOHA looks like:

default via 10.0.0.1 dev eth0
default via 10.0.1.1 dev eth1  metric 1
default via 10.0.2.1 dev eth2  metric 2
default via 10.0.3.1 dev eth3  metric 3
default via 10.0.4.1 dev eth4  metric 4

HAProxy configuration for Corporate website or ADFS proxies


These services are used by internet users only.

frontend ft_www
 bind 10.0.0.2:80
[...]

no need to specify any interface here, since the traffic comes from internet, HAProxy can let the kernel to use the default gateway which points in that direction (here eth0).

HAProxy configuration for Exchange 2010 or 2013


This service is used by both internal and internet users.

frontend ft_exchange
 bind 10.0.0.3:443
 bind 10.0.2.3:443 interface eth2
[...]

The responses to internet users will go through eth0 while the one for internal LAN users will use the default gateway configured on eth2 10.0.2.1.

HAProxy configuration for Sharepoint 2010 or 2013


This service is used by MPLS/VPN users and internal users.

frontend ft_exchange
 bind 10.0.1.4:443 interface eth1
 bind 10.0.2.4:443 interface eth2
[...]

The responses to MPLS/VPN users will go through eth1 default gateway 10.0.1.1 while the one for internal LAN users will use the default gateway configured on eth2 10.0.2.1.

Links

Posted in Aloha, architecture, HAProxy, layer7 | Tagged , , , , , | Leave a comment

How to protect application cookies while offloading SSL

SSL offloading

SSL offloading or acceleration is often seen as a huge benefit for applications. People usually forget that it may have impacts on the application itself. Some times ago, I wrote a blog article which lists these impacts and propose some solutions, using HAProxy.

One thing I forgot to mention at that time was Cookies.
You don’t want your clients to send their cookies (understand their identity) in clear through the Internet.
This is today’s article purpose.

Actually, there is a cookie attribute called Secure which can be emit by a server. When this attribute is set, the client SHOULD not send the cookie over a clear HTTP connection.

SSL offloading Diagram


Simple SSL offloading diagram:

|--------|              |---------|           |--------|
| client |  ==HTTPS==>  | HAProxy | --HTTP--> | Server |
|--------|              |---------|           |--------|

The client uses HTTPs to get connected on HAProxy, HAProxy gets connected to the application server through HTTP.

Even if HAProxy can forward client connection mode information to the application server, the application server may not protect its cookie…
Fortunately, we can use HAProxy for this purpose.

Howto make HAProxy to protect application cookie when SSL offloading is enabled

That’s the question.

The response is as simple as the configuration below:

acl https          ssl_fc
acl secured_cookie res.hdr(Set-Cookie),lower -m sub secure
rspirep ^(set-cookie:.*) \1;\ Secure if https !secured_cookie

The configuration above sets up the Secure attribute if it has not been setup by the application server while the client was browsing the application over a ciphered connection.

Related Links

Links

Posted in Uncategorized | Leave a comment

Emulating Active/passing application clustering with HAProxy

Synopsis

HAProxy is a Load-Balancer, this is a fact. It is used to route traffic to servers to primarily ensure applications reliability.

Most of the time, the sessions are locally stored in a server. Which means that if you want to split client traffic on multiple servers, you have to ensure each user can be redirected to the server which manages his session (if the server is available, of course). HAProxy can do this in many ways: we call it persistence.
Thanks to persistence, we usually says that any application can be load-balanced… Which is true in 99% of the cases. In very rare cases, the application can’t be load-balanced. I mean that there might be a lock somewhere in the code or for some other good reasons…

In such case, to ensure high-availability, we build “active/passive” clusters, where a node can be active at a time.
HAProxy can be use in different ways to emulate an active/passive clustering mode, and this is the purpose of today’s article.

Bear in mind that by “active/passive”, I mean that 100% of the users must be forwarded to the same server. And if a fail over occurs, they must follow it in the mean time!

Diagram

Let’s use one HAProxy with a couple of servers, s1 and s2.
When starting up, s1 is master and s2 is used as backup:

  -------------
  |  HAProxy  |
  -------------
   |         `
   |active    ` backup
   |           `
 ------       ------
 | s1 |       | s2 |
 ------       ------

Configuration

Automatic failover and failback

The configuration below makes HAProxy to use s1 when available, otherwise fail over to s2 if available:

defaults
 mode http
 option http-server-close
 timeout client 20s
 timeout server 20s
 timeout connect 4s

frontend ft_app
 bind 10.0.0.100:80 name app
 default_backend bk_app

backend bk_app
 server s1 10.0.0.1:80 check
 server s2 10.0.0.2:80 check backup

The most important keyword above is “backup” on s2 configuration line.
Unfortunately, as soon as s1 comes back, then all the traffic will fail back to it again, which can be acceptable for web applications, but not for active/passive

Automatic failover without failback

The configuration below makes HAProxy to use s1 when available, otherwise fail over to s2 if available.
When a failover has occured, no failback will be processed automatically, thanks to the stick table:

peers LB
 peer LB1 10.0.0.98:1234
 peer LB2 10.0.0.99:1234

defaults
 mode http
 option http-server-close
 timeout client 20s
 timeout server 20s
 timeout connect 4s

frontend ft_app
 bind 10.0.0.100:80 name app
 default_backend bk_app

backend bk_app
 stick-table type ip size 1 peers LB
 stick on dst
 server s1 10.0.0.1:80 check
 server s2 10.0.0.2:80 check backup

The stick table will maintain persistence based on destination IP address (10.0.0.100 in this case):

show table bk_app
# table: bk_app, type: ip, size:20480, used:1
0x869154: key=10.0.0.100 use=0 exp=0 server_id=1

With such configuration, you can trigger a fail back by disabling s2 during a few second period.

Links

Posted in Aloha, architecture, HAProxy | Tagged , , | 9 Comments

HAProxy advanced Redis health check

Introduction

Redis is an opensource nosql database working on a key/value model.
One interesting feature in Redis is that it is able to write data to disk as well as a master can synchronize many slaves.

HAProxy can load-balance Redis servers with no issues at all.
There is even a built-in health check for redis in HAProxy.
Unfortunately, there was no easy way for HAProxy to detect the status of a redis server: master or slave node. Hence people usually hacks this part of the architecture.

As written in the title of this post, we’ll learn today how to make a simple Redis infrastructure thanks to newest HAProxy advanced send/expect health checks.
This feature is available in HAProxy 1.5-dev20 and above.

Purpose is to make the redis infrastructure as simple as possible and ease fail over for the web servers. HAProxy will have to detect which node is MASTER and route all the connection to it.

Redis high availability diagram with HAProxy

Below, an ascii art diagram of HAProxy load-balancing Redis servers:

+----+ +----+ +----+ +----+
| W1 | | W2 | | W3 | | W4 |   Web application servers
+----+ +----+ +----+ +----+
     \     |   |     /
      \    |   |    /
       \   |   |   /
        +---------+
        | HAProxy |
        +---------+
           /   \
       +----+ +----+
       | R1 | | R2 |           Redis servers
       +----+ +----+

The scenario is simple:
  * 4 web application servers need to store and retrieve data to/from a Redis database
  * one (better using 2) HAProxy servers which load-balance redis connections
  * 2 (at least) redis servers in an active/standby mode with replication

Configuration

Below, is the HAProxy configuration for the

defaults REDIS
 mode tcp
 timeout connect  4s
 timeout server  30s
 timeout client  30s

frontend ft_redis
 bind 10.0.0.1:6379 name redis
 default_backend bk_redis

backend bk_redis
 option tcp-check
 tcp-check send PING\r\n
 tcp-check expect string +PONG
 tcp-check send info\ replication\r\n
 tcp-check expect string role:master
 tcp-check send QUIT\r\n
 tcp-check expect string +OK
 server R1 10.0.0.11:6379 check inter 1s
 server R2 10.0.0.12:6379 check inter 1s

The HAProxy health check sequence above allows to consider the Redis master server as UP in the farm and redirect connections to it.
When the Redis master server fails, the remaining nodes elect a new one. HAProxy will detect it thanks to its health check sequence.

It does not require third party tools and make fail over transparent.

Links

Posted in architecture, HAProxy, optimization | Tagged | 1 Comment

failover and worst case management with HAProxy

Synopsis

One of HAProxy strength is that it is flexible and allows to redirect traffic based on events and internal status.
In the current article, I’ll show how HAProxy can be useful to handle traffic when worst cases happen.

By worst case I mean the moment when something went wrong in your architecture and your application because partially or totally unavailable.

Cases

Backup server

When all servers in a farm are down, we want to redirect traffic to a backup server which delivers either sorry pages or a degraded mode of the application.
This can be done easily in HAProxy by adding the keyword backup on the server line. If multiple backup servers are configured, only the first active one is used.

Below, the HAProxy configuration corresponding to this case:

frontent ft_app
 bind 10.0.0.1:80
 default_backend bk_app_main

backend bk_app_main
 server s1 10.0.0.101:80 check
 server s2 10.0.0.102:80 check
 server s3 10.0.0.103:80 check backup
 server s4 10.0.0.104:80 check backup

In this case, s3 will be used first, until it fails, then s4 will be used.

Multiple backup servers

In some cases, when the farm takes a huge traffic, we may want to use many backup servers at a time. This can be achieved by enabling the option allbackups in HAProxy configuration.

Below, the HAProxy configuration corresponding to this case:

frontent ft_app
 bind 10.0.0.1:80
 default_backend bk_app_main

backend bk_app_main
 option allbackups
 server s1 10.0.0.101:80 check
 server s2 10.0.0.102:80 check
 server s3 10.0.0.103:80 check backup
 server s4 10.0.0.104:80 check backup

In this case, both s3 and s4 will be used if they are available.

Farm failover

Despite the case above improves a bit our failover scenario, it has some weaknesses. For example we must wait until all the production servers are DOWN before using the backup servers.
HAProxy can failover traffic to a backup farm when the main one has not enough capacity or, worst case, no capacity anymore.

Below, the HAProxy configuration corresponding to this case:

frontent ft_app
 bind 10.0.0.1:80

# detect capacity issues in production farm
 acl MAIN_not_enough_capacity nb_srv(bk_app_main) le 2
# failover traffic to backup farm
 use_backend bk_app_backup if MAIN_not_enough_capacity

 default_backend bk_app_main

backend bk_app_main
 server s11 10.0.0.101:80 check
 server s12 10.0.0.102:80 check
 server s13 10.0.0.103:80 check
 server s14 10.0.0.104:80 check

backend bk_app_backup
 server s21 20.0.0.101:80 check
 server s22 20.0.0.102:80 check

Farm failover with backup servers

Of course, we could combine all the options above.
First we want to failover to a backup farm if the production one has not enough capacity, second, we want to use 2 backup servers when all the production servers from the backup farm are DOWN.

Below, the HAProxy configuration corresponding to this case:

frontent ft_app
 bind 10.0.0.1:80

# detect capacity issues in production farm
 acl MAIN_not_enough_capacity nb_srv(bk_app_main) le 2
# failover traffic to backup farm
 use_backend bk_app_backup if MAIN_not_enough_capacity

 default_backend bk_app_main

backend bk_app_main
 server s11 10.0.0.101:80 check
 server s12 10.0.0.102:80 check
 server s13 10.0.0.103:80 check
 server s14 10.0.0.104:80 check

backend bk_app_backup
 option allbackups
 server s21 20.0.0.101:80 check
 server s22 20.0.0.102:80 check
 server s23 20.0.0.103:80 check backup
 server s24 20.0.0.104:80 check backup

Worst case: no servers available anymore

Well, imagine you plugged all your servers on a single switch, HAProxy box has 2 interfaces, one on the public switch, on on the server switch. Of course, this is not how you plugged your servers, don’t you?
Imagine the server switch fails, then no servers are available anymore. Then HAProxy can be used to deliver sorry pages for you.

Below, the HAProxy configuration corresponding to this case:

frontent ft_app
 bind 10.0.0.1:80

# sorry page to return when worst case happens
 errorfile 503 /etc/haproxy/errorfiles/sorry.http

# detect capacity issues in production farm
 acl MAIN_not_enough_capacity nb_srv(bk_app_main) le 2
# failover traffic to backup farm
 use_backend bk_app_backup if MAIN_not_enough_capacity

 default_backend bk_app_main

backend bk_app_main
 server s11 10.0.0.101:80 check
 server s12 10.0.0.102:80 check
 server s13 10.0.0.103:80 check
 server s14 10.0.0.104:80 check

backend bk_app_backup
 option allbackups
 server s21 20.0.0.101:80 check
 server s22 20.0.0.102:80 check
 server s23 20.0.0.103:80 check backup
 server s24 20.0.0.104:80 check backup

And below, the content of the sorry.http page:

HTTP/1.0 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html>
<body>
<h1>Sorry page</h1>
Sorry, we're under maintenance
</body>
</html>

Important notes


Health checking


Health checking must be enabled on the servers. Without health checking, HAProxy can’t know the server status and then can’t decide to failover traffic.

Persistence


If a persistence information points to one backup server, then HAProxy will keep on using it, even if production servers are available.

Links

Posted in Uncategorized | 5 Comments

Configuring HAProxy and Nginx for SPDY

Introduction to SPDY / HTTP-bis

SPDY is a protocol designed by google which aims to fix HTTP/1.1 protocol weaknesses and to adapt this 14 years old protocol to today’s internet devices and requirements.
Back in 1999, when HTTP/1.1 was designed, there was no mobile devices, the web pages were composed by HTML with a few images, almost no javascript and no CSS. The ISP delivered internet over very slow connections.
HTTP/2.0 has to address today’s and tomorrow’s need when much more devices can browse the internet from very different type of connections (very slow with high packet loss or very fast ones) and more and more people want multimedia content and interactivity. Websites became “web applications”.

SPDY is not HTTP 2.0! But HTTP 2.0 will use SPDY as basement

Note that as long as HTTP/2.0 has not been released officially, ALL articles written on SPDY, NPN, ALPN, etc may be outdated at some point.
Sadly, this is true for the present article, BUT I’ll try to keep it up to date :)
As an example, NPN is a TLS extension that has been designed to allow a client and a server to negotiate the protocol which will be used at the above network layer (in our case this is application layer).
Well, this NPN TLS extension is going to be outdated soon by an official RFC called “Transport Layer Security (TLS) Application Layer Protocol Negotiation Extension” (shortened to ALPN).

Also, we’re already at the third version of SPDY protocol (3.1 to be accurate). It changes quite often.

HAProxy and SPDY

As Willy explained on HAProxy‘s mailing list, HAProxy won’t implement SPDY.
But, as soon as HTTP/2.0 will be released officially, then the development will focus on it.
Main driver for this is the problem seen in introduction: waste of time because the drafts are updated quite often.

Saying that, HAProxy can be used to:
* load-balance SPDY web servers
* detect SPDY protocol through NPN (or more recent ALPN) TLS extension

Diagram

The picture below shows how HAProxy can be used to detect and split HTTP and SPDY traffic over an HTTPs connection:
spdy

Basically, HAProxy uses the NPN (and later the ALPN) TLS extension to figure out whether the client can browse the website using SPDY. If yes, the connection is forwarded to the SPDY farm (here hosted on nginx), otherwise, the connection is forwarded to the HTTP server farm (here hosted on nginx too).

Configuration

HAProxy configuration example for NPN and SPDY


Below, the HAProxy configuration to detect and split HTTP and SPDY traffic to two different farms:

defaults
 mode tcp
 log global
 option tcplog
 timeout connect           4s
 timeout server          300s
 timeout client          300s

frontend ft_spdy
 bind 64.210.140.163:443 name https ssl crt STAR_haproxylab_net.pem npn spdy/2

# tcp log format + SSL information (TLS version, cipher in use, SNI, NPN)
 log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ {%sslv/%sslc/%[ssl_fc_sni]/%[ssl_fc_npn]}

# acls: npn
 acl npn_spdy           ssl_fc_npn -i spdy/2

# spdy redirection
 use_backend bk_spdy      if npn_spdy

 default_backend bk_http

backend bk_spdy
 option httpchk HEAD /healthcheck
 server nginx 127.0.0.1:8082 maxconn 100 check port 8081

backend bk_http
 option httpchk HEAD /healthcheck
 http-request set-header Spdy no
 server nginx 127.0.0.1:8081 maxconn 100 check

NGINX configuration example for SPDY


And the corresponding nginx configuration:
Note that I use a LUA script to check if the “Spdy: no” HTTP header is present.
  * if present: then the connection was made over HTTP in HAProxy
  * if not present, then the connection was made over SPDY

   server {
      listen 127.0.0.1:8081;
      listen 127.0.0.1:8082 spdy;
      root /var/www/htdocs/spdy/;

        location = /healthcheck {
          access_log off;
          return 200;
        }

        location / {
             default_type 'text/plain';
             content_by_lua '
               local c = ngx.req.get_headers()["Spdy"];
               if c then
                 ngx.say("Currently browsing over HTTP")
               else
                 ngx.say("Currently browsing over spdy")
               end
                return
             ';
        }

   }

Testing


This setup is in production.
Simply browse https://spdy.haproxylab.net/ to give it a try.

Related links

Links

Posted in architecture, HAProxy, layer7, ssl | Tagged , , , , | Leave a comment

Howto transparent proxying and binding with HAProxy and ALOHA Load-Balancer

Transparent Proxy

HAProxy works has a reverse-proxy. Which means it maintains 2 connections when allowing a client to cross it:
  - 1 connection between HAProxy and the client
  - 1 connection between HAProxy and the server

HAProxy then manipulate buffers between these two connections.
One of the drawback of this mode is that HAProxy will let the kernel to establish the connection to the server. The kernel is going to use a local IP address to do this.
Because of this, HAProxy “hides” the client IP by its own one: this can be an issue in some cases.
Here comes the transparent proxy mode: HAProxy can be configured to spoof the client IP address when establishing the TCP connection to the server. That way, the server thinks the connection comes from the client directly (of course, the server must answer back to HAProxy and not to the client, otherwise it can’t work: the client will get an acknowledge from the server IP while it has established the connection on HAProxy‘s IP).

Transparent binding

By default, when one want HAProxy to get traffic, we have to tell it to bind an IP address and a port.
The IP address must exist on the operating system (unless you have setup the sysctl net.ipv4.ip_nonlocal_bind) and the OS must announce the availability to the other devices on the network through ARP protocol.
Well, in some cases we want HAProxy to be able to catch traffic on the fly without configuring any IP address or VRRP or whatever…
This is where transparent binding comes in: HAProxy can be configured to catch traffic on the fly even if the destination IP address is not configured on the server.
These IP addresses will never be pingable, but they’ll deliver the services configured in HAProxy.

HAProxy and the Linux Kernel

Unfortunately, HAProxy can’t do transparent binding or proxying alone. It must stand on a compiled and tuned Linux Kernel and operating system.
Below, I’ll explain how to do this in a standard Linux distribution.
Here is the check list to meet:
  1. appropriate HAProxy compilation option
  2. appropriate Linux Kernel compilation option
  3. sysctl settings
  4. iptables rules
  5. ip route rules
  6. HAProxy configuration

HAProxy compilation requirements


First of all, HAProxy must be compiled with the option TPROXY enabled.
It is enabled by default when you use the target LINUX26 or LINUX2628.

Linux Kernel requirements

You have to ensure your kernel has been compiled with the following options:
  - CONFIG_NETFILTER_TPROXY
  - CONFIG_NETFILTER_XT_TARGET_TPROXY

Of course, iptables must be enabled as well in your kernel :)

sysctl settings

The following sysctls must be enabled:
  - net.ipv4.ip_forward
  - net.ipv4.ip_nonlocal_bind

iptables rules


You must setup the following iptables rules:

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT

Purpose is to mark packets which matches a socket bound locally (by HAProxy).

IP route rules


Then, tell the Operating System to forward packets marked by iptables to the loopback where HAProxy can catch them:

ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

HAProxy configuration


Finally, you can configure HAProxy.
  * Transparent binding can be configured like this:

[...]
frontend ft_application
  bind 1.1.1.1:80 transparent
[...]

  * Transparent proxying can be configured like this:

[...]
backend bk_application
  source 0.0.0.0 usesrc clientip
[...]

Transparent mode in the ALOHA Load-Balancer


Now, the same steps in the ALOHA Load-balancer, which is an HAProxy based load-balancing appliance:
  1-5. not required, the ALOHA kernel is deeply tuned for this purpose
  6. HAProxy configuration

LB Admin tab (AKA click mode)

  * Transparent binding can be configured like this, when editing a Frontend listener:
frontend_listener_transparent

  * Transparent proxying can be configured like this when editing a farm:
backend_transparent

LB Layer 7 tab (vi in a browser mode)


  * Transparent binding can be configured like this:

[...]
frontend ft_application
  bind 1.1.1.1:80 transparent
[...]

  * Transparent proxying can be configured like this:

[...]
backend bk_application
  source 0.0.0.0 usesrc clientip
[...]

Links

Posted in Aloha, architecture, HAProxy, layer7 | Tagged , , , , | Leave a comment