This one is so easy I don’t know why I haven’t gotten around to it sooner. It took my dev to be all, “OMG! The servers! How will we do a rolling deploy without impacting uptime!1!!eleven!” Thankfully, Amazon is there with Elastic Load Balancing (ELB) so I didn’t really need to get off my ass and be excited. Amazon Web Services: Doing the hard work so you don’t have to.
Before we dive in, a little background on operations. We’ve been making use of the elastic IP feature since it came out along with round robin DNS to handle the distribution of traffic over the web servers. Sure, it is crude and primitive but it works and I am a big fan of implementing the simplest possible solution since more moving parts results in exponentially more headaches. And, yes, we tried the HAProxy and the Nginx route but the cost+plus heartburn factor was too high for what we needed. Anyway, things have worked just fine with a minimum of effort on our part but our needs are changing so we need a solution that is both flexible and forgiving. Brittle is only good with peanuts.
First things first, create an ELB…
elb-create-lb eeniemeenie --headers --listener "lb-port=443,instance-port=8443,protocol=TCP" \
--listener "lb-port=80,instance-port=80,protocol=http" --availability-zones us-east-1cNow you might be saying to yourself, “WTF, James. What is with the 443 => 8443 when 80 => 80?” Simple: “Currently, Elastic Load Balancing does not have SSL termination capability.” That basically means your application servers need to handle the SSL part but honestly this is fine because the way ELB is engineered the traffic between the balancer and your instances outside the firewall so you’ll want that traffic passing over SSL anyway.
Next, configure health checks…
elb-configure-healthcheck eeniemeenie --headers --target "TCP:8443" --interval 30 --timeout 20 \
--unhealthy-threshold 2 --healthy-threshold 2
elb-configure-healthcheck eeniemeenie --headers --target "HTTP:80/up.html" --interval 10 --timeout 5 \
--unhealthy-threshold 2 --healthy-threshold 2
These two health checks are intended to do different things. The one for TCP:8443 is to see if our application is up and serving (we force all traffic over SSL) while the latter is intended for rolling deploys, when we begin a deploy we’ll rename that file with the intention that the balancer sees it go missing and pulls the instance out of the pool. When everything is done and the deploy was successful we plop up.html back in and the balancer throws the instance back in the pool.
On the Apache side of things I decided to do a little mod_rewrite mumbo jumbo to deal with the new up.html file. Since I want all application related traffic forced to SSL the monitoring of that file by ELB presents a little wrinkle to my normal sledgehammer approach to things. Easily addressed though:
RewriteEngine on
RewriteCond %{REQUEST_URI} !(up.html)
RewriteRule ^(.*)$ https://%{SERVER_NAME}$1 [L,R]Now, because I am a grumpy bastard I really don’t want up.html to be available over SSL.
RewriteEngine on
RewriteCond %{REQUEST_URI} (up.html)
RewriteRule ^(.*)$ http://%{SERVER_NAME}$1 [L,R]That is basically it. Mind you, I have not punched this down for production but our initial tests have been very promising, in particular the rolling deployment feature.
(Parts of this solution were gratefully cribbed from Serk and Lead Thinking as well as the ever popular RTFM)
Comments
James, Dale
james, Mike
james, Mike, james [...]
james, Mike
james, Mike
james, Kyle Daigle