This was probably one of the more radical switches in architecture that we’ve made in the recent past. For the past 7 months we have been successfully running Apache + mod_proxy + mongrel with some limited PHP applications bolted on but the whole set up felt a tad bloated and a little more than unstable as we tested various scaling scenarios. With the rails community chatting about the hotness that is thin, nginx, and HAProxy we decided to see what it would take to migrate.
The catch with our infrastructure though is that we have broken apart our static assets from rails so the usual localhost simplicity isn’t there which, unfortunately, is how most of the tutorials are aimed at. In our case, the application sits in a pool of servers and one of the things that we wanted to do was leverage HAProxy to balance each nginx instance over a group of primary and secondary application servers with the primary and secondary status staggered between each nginx instance. Igvita’s post was the inspiration for this and our goal is to create a more fault tolerant environment built on shared services rather than our current setup of largely discrete stacks.
The first thing I tackled was setting up nginx by breaking apart the rails application and any PHP applications into separate virtual hosts. First up is the rails config…
upstream thin {
server 127.0.0.1:8700;
}
server {
listen 80;
server_name first.server.name;
rewrite ^/(.*) https://what.ever.you.want/$1 permanent;
}
server {
listen 443;
ssl on;
ssl_session_timeout 5m;
ssl_protocols SSLv2 SSLv3 TLSv1;
ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers on;
# path to your certificate
# if you have an intermediate cert then you need to add the contents to the end of the cert file
ssl_certificate /where/your/cert/is.pem;
# path to your ssl key
ssl_certificate_key /where/your/key/is.key;
# standard rails configuration goes here.
root /location/of/your/site/root;
# rewrite_log on;
if (-f $document_root/system/maintenance.html) {
rewrite ^(.*)$ /system/maintenance.html last;
break;
}
location ~ ^/$ {
if (-f /index.html){
rewrite (.*) /index.html last;
}
proxy_pass http://thin;
}
location / {
if (!-f $request_filename.html) {
proxy_pass http://thin;
}
rewrite (.*) $1.html last;
}
location ~ .html {
root /location/of/your/site/root;
}
location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|pdf|txt|js|mov)$ {
root /location/of/your/site/root;
}
location / {
proxy_pass http://thin;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-FORWARDED_PROTO https;
}
}
And our PHP config…
server {
### PHP Support ###
listen 80;
server_name second.server.name;
access_log /location/of/your/site/root/logs/blog-access.log;
error_log /location/of/your/site/root/logs/blog-error.log;
if (!-e $request_filename) {
rewrite ^([_0-9a-zA-Z-]+)?(/wp-.*) $2 last;
rewrite ^([_0-9a-zA-Z-]+)?(/.*\.php)$ $2 last;
rewrite ^ /index.php last;
}
location / {
root / /location/of/your/site/root;
index index.html index.php index.htm;
}
location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|pdf|txt|js|mov)$ {
root /location/of/your/site/root;
}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_FILENAME /location/of/your/site/root/$fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
}
}
Next up is the HAProxy configuration…
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
nbproc 1
pidfile /var/run/haproxy.pid
#debug
#quiet
user haproxy
group haproxy
defaults
log global
mode http
option httplog
option dontlognull
retries 15
redispatch
contimeout 60000
clitimeout 150000
srvtimeout 60000
option httpclose # disable keepalive (HAProxy does not yet support the HTTP keep-alive mode)
option abortonclose # enable early dropping of aborted requests from pending queue
option httpchk # enable HTTP protocol to check on servers health
listen thin *:8700
option httpchk
mode http
option forwardfor except 127.0.0.1/8
balance roundrobin
server web01 hostname-of-server:8100 weight 1 minconn 1 maxconn 6 check inter 40000
etc....
There are a couple of things to note here: to get HAProxy to fetch content from servers other than localhost you’ll need to chuck in a wildcard: listen thin *:8700, and to get logging running you’ll need to edit /etc/syslog.conf adding the following lines:
# Save HA-Proxy logs
local0.* /var/log/haproxy_0.log
local1.* /var/log/haproxy_1.log
As well as edit /etc/default/syslogd:
# For remote UDP logging use SYSLOGD="-r"
SYSLOGD="-r"
One last thing that drove me almost to the brink of madness is that HAProxy, at least in the build on Ubuntu 8.04, is finicky about how the configuration file is laid out. Each section default, global, and listen has to have the parameters defined with a tab preceding each and while HAProxy would start and accept request from nginx with anything else it would not fetch from the thin server pool.
So that is our front-end, what about the application pool? Turns out that Thin is just as easy to set up as a mongrel cluster and only took a minimum of effort on our part to get it dialed in with God and serving upstream. We edited the stock init script to reflect where we store the yamls and massaged God for the changes in clustering.
Here’s our init script:
#!/bin/sh
### BEGIN INIT INFO
# Provides: thin
# Required-Start: $local_fs $remote_fs
# Required-Stop: $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: S 0 1 6
# Short-Description: thin initscript
# Description: thin
### END INIT INFO
# Original author: Forrest Robertson
# Do NOT "set -e"
DAEMON=/usr/bin/thin
SCRIPT_NAME=/etc/init.d/thin
CONFIG_PATH=/location/of/your/yamls
# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0
case "$1" in
start)
$DAEMON start --all $CONFIG_PATH
;;
stop)
$DAEMON stop --all $CONFIG_PATH
;;
restart)
$DAEMON restart --all $CONFIG_PATH
;;
*)
echo "Usage: $SCRIPT_NAME {start|stop|restart}" >&2
exit 3
;;
esac
:
And here’s a sample yaml:
---
user: user-which-runs
group: group-which-runs
chdir: /location/of/your/app
log: log/thin.log
port: 8100
environment: staging
pid: /location/of/your/pids.pid
servers: 3
God is very similar to what we had been running with a mongrel cluster:
RAILS_ROOT = "/location/of/your/app"
%w{8100 8101 8102}.each do |port|
God.watch do |w|
w.group = 'pack_01'
w.name = "thin-#{port}"
w.interval = 30.seconds # default
w.start = "thin start -C /location/of/your.yaml -o #{port}"
w.stop = "thin stop -C /location/of/your.yaml -o #{port}"
w.restart = "thin stop -C/location/of/your.yaml -o #{port} && thin start -C /location/of/your.yaml -o #{port}"
w.start_grace = 15.seconds
w.restart_grace = 15.seconds
w.pid_file = "/location/of/your/pids.#{port}.pid"
w.behavior(:clean_pid_file)
w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
end
end
w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.above = 150.megabytes
c.times = [3, 5] # 3 out of 5 intervals
end
restart.condition(:cpu_usage) do |c|
c.above = 50.percent
c.times = 5
end
end
# lifecycle
w.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minute
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 2.hours
end
end
end
end
There you have it, a completely rebuilt stack leveraging lean, fast, and stable services.
Gratefully cribbed from HowtoForge, John Yerhot, and Igvita.
