This one is so easy I don’t know why I haven’t gotten around to it sooner. It took my dev to be all, “OMG! The servers! How will we do a rolling deploy without impacting uptime!1!!eleven!” Thankfully, Amazon is there with Elastic Load Balancing (ELB) so I didn’t really need to get off my ass and be excited. Amazon Web Services: Doing the hard work so you don’t have to.
Before we dive in, a little background on operations. We’ve been making use of the elastic IP feature since it came out along with round robin DNS to handle the distribution of traffic over the web servers. Sure, it is crude and primitive but it works and I am a big fan of implementing the simplest possible solution since more moving parts results in exponentially more headaches. And, yes, we tried the HAProxy and the Nginx route but the cost+plus heartburn factor was too high for what we needed. Anyway, things have worked just fine with a minimum of effort on our part but our needs are changing so we need a solution that is both flexible and forgiving. Brittle is only good with peanuts.
First things first, create an ELB…
elb-create-lb eeniemeenie --headers --listener "lb-port=443,instance-port=8443,protocol=TCP" \
--listener "lb-port=80,instance-port=80,protocol=http" --availability-zones us-east-1cNow you might be saying to yourself, “WTF, James. What is with the 443 => 8443 when 80 => 80?” Simple: “Currently, Elastic Load Balancing does not have SSL termination capability.” That basically means your application servers need to handle the SSL part but honestly this is fine because the way ELB is engineered the traffic between the balancer and your instances outside the firewall so you’ll want that traffic passing over SSL anyway.
Next, configure health checks…
elb-configure-healthcheck eeniemeenie --headers --target "TCP:8443" --interval 30 --timeout 20 \
--unhealthy-threshold 2 --healthy-threshold 2
elb-configure-healthcheck eeniemeenie --headers --target "HTTP:80/up.html" --interval 10 --timeout 5 \
--unhealthy-threshold 2 --healthy-threshold 2
These two health checks are intended to do different things. The one for TCP:8443 is to see if our application is up and serving (we force all traffic over SSL) while the latter is intended for rolling deploys, when we begin a deploy we’ll rename that file with the intention that the balancer sees it go missing and pulls the instance out of the pool. When everything is done and the deploy was successful we plop up.html back in and the balancer throws the instance back in the pool.
On the Apache side of things I decided to do a little mod_rewrite mumbo jumbo to deal with the new up.html file. Since I want all application related traffic forced to SSL the monitoring of that file by ELB presents a little wrinkle to my normal sledgehammer approach to things. Easily addressed though:
RewriteEngine on
RewriteCond %{REQUEST_URI} !(up.html)
RewriteRule ^(.*)$ https://%{SERVER_NAME}$1 [L,R]Now, because I am a grumpy bastard I really don’t want up.html to be available over SSL.
RewriteEngine on
RewriteCond %{REQUEST_URI} (up.html)
RewriteRule ^(.*)$ http://%{SERVER_NAME}$1 [L,R]That is basically it. Mind you, I have not punched this down for production but our initial tests have been very promising, in particular the rolling deployment feature.
(Parts of this solution were gratefully cribbed from Serk and Lead Thinking as well as the ever popular RTFM)
Tags: Apache, EC2, load balancing
Very interesting – thanks for sharing this cool technique!
One issue: An ELB only supports a single health check at a time. So your second health check for up.html overrides the TCP:8443 health check. Here’s the results when I try these commands on my ELB testLB:
shlomo$ elb-create-lb testLB –listener “lb-port=80,instance-port=80,protocol=http” –listener “lb-port=443,instance-port=8443,protocol=tcp” –availability-zones us-east-1a
DNS-NAME testLB-1819819914.us-east-1.elb.amazonaws.com
shlomo$ elb-configure-healthcheck testLB –headers –target “HTTP:80/status” –interval 5 –timeout 3 –unhealthy-threshold 2 –healthy-threshold 2
HEALTH-CHECK TARGET INTERVAL TIMEOUT HEALTHY-THRESHOLD UNHEALTHY-THRESHOLD
HEALTH-CHECK HTTP:80/status 5 3 2 2
shlomo$ elb-configure-healthcheck testLB –headers –target “TCP:8443″ –interval 10 –timeout 3 –unhealthy-threshold 2 –healthy-threshold 2
HEALTH-CHECK TARGET INTERVAL TIMEOUT HEALTHY-THRESHOLD UNHEALTHY-THRESHOLD
HEALTH-CHECK TCP:8443 10 3 2 2
shlomo$ elb-describe-lbs –show-xml
testLB
testLB-1819819914.us-east-1.elb.amazonaws.com
HTTP
80
80
TCP
443
8443
us-east-1a
TCP:8443
10
3
2
2
2009-11-23T21:44:24.420Z
795ddb2e-d879-11de-b139-d72195628521
Only the second health check is retained.
Aw, crud – the XML tags got munged. How about this:
shlomo$ elb-describe-lbs –show-long
LOAD-BALANCER,testLB,testLB-1819819914.us-east-1.elb.amazonaws.com,”{interval=10,target=TCP:8443,timeout=3,healthy-threshold=2,unhealthy-threshold=2}”,us-east-1a,(nil),”{protocol=HTTP,lb-port=80,instance-port=80, protocol=TCP,lb-port=443,instance-port=8443}”,2009-11-23T21:44:24.420Z
Aw, fiddlesticks. Well, that puts another wrinkle in testing. I’ll have loop back and wrestle with that a little more as I *think* it is entirely possible for Passenger to crap the bed but Apache keeps on happily serving files. :-/