This was probably one of the more radical switches in architecture that we’ve made in the recent past.  For the past 7 months we have been successfully running Apache + mod_proxy + mongrel with some limited PHP applications bolted on but the whole set up felt a tad bloated and a little more than unstable as we tested various scaling scenarios.  With the rails community chatting about the hotness that is thin, nginx, and HAProxy we decided to see what it would take to migrate.

The catch with our infrastructure though is that we have broken apart our static assets from rails so the usual localhost simplicity isn’t there which, unfortunately, is how most of the tutorials are aimed at.  In our case, the application sits in a pool of servers and one of the things that we wanted to do was leverage HAProxy to balance each nginx instance over a group of primary and secondary application servers with the primary and secondary status staggered between each nginx instance. Igvita’s post was the inspiration for this and our goal is to create a more fault tolerant environment built on shared services rather than our current setup of largely discrete stacks.

The first thing I tackled was setting up nginx by breaking apart the rails application and any PHP applications into separate virtual hosts. First up is the rails config…

upstream thin {
server 127.0.0.1:8700;
}

server {
listen       80;
server_name  first.server.name;
rewrite ^/(.*) https://what.ever.you.want/$1 permanent;
}

server {
listen 443;
ssl on;
ssl_session_timeout  5m;
ssl_protocols  SSLv2 SSLv3 TLSv1;
ssl_ciphers  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers   on;

# path to your certificate
# if you have an intermediate cert then you need to add the contents to the end of the cert file
ssl_certificate /where/your/cert/is.pem;

# path to your ssl key
ssl_certificate_key /where/your/key/is.key;

# standard rails configuration goes here.
root /location/of/your/site/root;

#        rewrite_log on;

if (-f $document_root/system/maintenance.html) {
rewrite  ^(.*)$  /system/maintenance.html last;
break;
}

location ~ ^/$ {
if (-f /index.html){
rewrite (.*) /index.html last;
}
proxy_pass  http://thin;
}

location / {
if (!-f $request_filename.html) {
proxy_pass  http://thin;
}
rewrite (.*) $1.html last;
}

location ~ .html {
root /location/of/your/site/root;
}

location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|pdf|txt|js|mov)$ {
root  /location/of/your/site/root;
}

location / {
proxy_pass  http://thin;
proxy_redirect     off;
proxy_set_header   Host             $host;
proxy_set_header   X-Real-IP        $remote_addr;
proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
proxy_set_header X-FORWARDED_PROTO https;
}
}

And our PHP config…

server {
### PHP Support ###
listen       80;
server_name  second.server.name;
access_log  /location/of/your/site/root/logs/blog-access.log;
error_log  /location/of/your/site/root/logs/blog-error.log;

if (!-e $request_filename) {
rewrite ^([_0-9a-zA-Z-]+)?(/wp-.*) $2 last;
rewrite ^([_0-9a-zA-Z-]+)?(/.*\.php)$ $2 last;
rewrite ^ /index.php last;
}

location / {
root / /location/of/your/site/root;
index index.html index.php index.htm;
}

location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|pdf|txt|js|mov)$ {
root /location/of/your/site/root;
}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000

location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_FILENAME  /location/of/your/site/root/$fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
}
}

Next up is the HAProxy configuration…

global
	log 127.0.0.1	local0
	log 127.0.0.1	local1 notice
	nbproc		1
	pidfile		/var/run/haproxy.pid
	#debug
	#quiet
	user haproxy
	group haproxy

defaults
	log		global
	mode		http
	option		httplog
	option		dontlognull
	retries		15
	redispatch
	contimeout	60000
	clitimeout	150000
	srvtimeout	60000
	option          httpclose     # disable keepalive (HAProxy does not yet support the HTTP keep-alive mode)
	option          abortonclose  # enable early dropping of aborted requests from pending queue
	option          httpchk       # enable HTTP protocol to check on servers health

listen	thin *:8700
	option httpchk
        mode http
        option forwardfor except 127.0.0.1/8
	balance roundrobin
        server web01 hostname-of-server:8100 weight 1 minconn 1 maxconn 6 check inter 40000
        etc....

There are a couple of things to note here: to get HAProxy to fetch content from servers other than localhost you’ll need to chuck in a wildcard: listen thin *:8700, and to get logging running you’ll need to edit /etc/syslog.conf adding the following lines:

# Save HA-Proxy logs
	local0.*                                                /var/log/haproxy_0.log
	local1.*                                                /var/log/haproxy_1.log

As well as edit /etc/default/syslogd:

# For remote UDP logging use SYSLOGD="-r"
SYSLOGD="-r"

One last thing that drove me almost to the brink of madness is that HAProxy, at least in the build on Ubuntu 8.04, is finicky about how the configuration file is laid out. Each section default, global, and listen has to have the parameters defined with a tab preceding each and while HAProxy would start and accept request from nginx with anything else it would not fetch from the thin server pool.

So that is our front-end, what about the application pool? Turns out that Thin is just as easy to set up as a mongrel cluster and only took a minimum of effort on our part to get it dialed in with God and serving upstream. We edited the stock init script to reflect where we store the yamls and massaged God for the changes in clustering.

Here’s our init script:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          thin
# Required-Start:    $local_fs $remote_fs
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      S 0 1 6
# Short-Description: thin initscript
# Description:       thin
### END INIT INFO

# Original author: Forrest Robertson

# Do NOT "set -e"

DAEMON=/usr/bin/thin
SCRIPT_NAME=/etc/init.d/thin
CONFIG_PATH=/location/of/your/yamls

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

case "$1" in
  start)
	$DAEMON start --all $CONFIG_PATH
	;;
  stop)
	$DAEMON stop --all $CONFIG_PATH
	;;
  restart)
	$DAEMON restart --all $CONFIG_PATH
	;;
  *)
	echo "Usage: $SCRIPT_NAME {start|stop|restart}" >&2
	exit 3
	;;
esac

:

And here’s a sample yaml:

---
user: user-which-runs
group: group-which-runs
chdir: /location/of/your/app
log: log/thin.log
port: 8100
environment: staging
pid: /location/of/your/pids.pid
servers: 3

God is very similar to what we had been running with a mongrel cluster:

RAILS_ROOT = "/location/of/your/app"

%w{8100 8101 8102}.each do |port|
 God.watch do |w|
    w.group = 'pack_01'
    w.name = "thin-#{port}"
    w.interval = 30.seconds # default
    w.start = "thin start -C /location/of/your.yaml -o #{port}"
    w.stop = "thin stop -C /location/of/your.yaml -o #{port}"
    w.restart = "thin stop -C/location/of/your.yaml -o #{port} && thin start -C /location/of/your.yaml -o #{port}"
    w.start_grace = 15.seconds
    w.restart_grace = 15.seconds
    w.pid_file = "/location/of/your/pids.#{port}.pid"

    w.behavior(:clean_pid_file)

    w.start_if do |start|
      start.condition(:process_running) do |c|
        c.interval = 5.seconds
        c.running = false
      end
    end

    w.restart_if do |restart|
      restart.condition(:memory_usage) do |c|
        c.above = 150.megabytes
        c.times = [3, 5] # 3 out of 5 intervals
      end

      restart.condition(:cpu_usage) do |c|
        c.above = 50.percent
        c.times = 5
      end
    end

    # lifecycle
    w.lifecycle do |on|
      on.condition(:flapping) do |c|
        c.to_state = [:start, :restart]
        c.times = 5
        c.within = 5.minute
        c.transition = :unmonitored
        c.retry_in = 10.minutes
        c.retry_times = 5
        c.retry_within = 2.hours
      end
    end
  end
end

There you have it, a completely rebuilt stack leveraging lean, fast, and stable services.

Gratefully cribbed from HowtoForge,  John Yerhot, and  Igvita.