Posts Tagged ‘Rails’

Continuous Integration Testing, CruiseControl.rb, Github, Apache, and You!

Friday, March 20th, 2009

“Ain’t no test like an elwoodicious test ’cause an elwoodicious test don’t pass!”

So one one of my co-workers thought it would be an awesome idea if we were continually reminded of our shortcomings regarding tests so he passed along a link to the CruiseControl.rb project. Getting up an running is a fairly simple affair, particularly if you are just running it locally, but we wanted to drop this on our QA server and have the dashboard served up via Apache.  Not too difficult, just a couple more steps.

Grabbing the code is the easy part, git clone git://github.com/thoughtworks/cruisecontrol.rb.git.

Add your project, ./cruise add Project-Name -s git -b branch-name-here -r git@github.com:username/project-name.git

Set up the configuration for your projects by editing ~/.cruise/site_config.rb.  The file is largely self-explanatory but it would be helpful to set Configuration.dashboard_url and Configuration.default_polling_interval which are nested in the middle of the file.

Set up the defaults for each project, like getting emails and polling times, in ~/.cruise/projects/project-name/cruise_config.rb.

Now the fun part, we use Apache with passenger so the tact we took was to install mongrel and configure Apache’s balancer to look for it on a specified port as well as password protect the app with basic authentication:

Edit your site file and add a virtual host…

<VirtualHost *:80>
ServerName cruise.mydomain.com
ServerAlias cruise.mydomain.com

Include /etc/apache2/sites-available/app.cruise

</VirtualHost>

Then create the app.cruise file with the following content…

ServerSignature Off

<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>

DocumentRoot /where/you/put/cruisecontrol.rb/public

<Directory “/where/you/put/cruisecontrol.rb/public”>
Options FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
AuthName “Secure Area”
AuthType Basic
AuthUserFile
/where/you/put/.htpasswd
require valid-user
</Directory>

RewriteEngine On

# Redirect all non-static requests to cluster
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://mongrel_cluster%{REQUEST_URI} [P,QSA,L]

# Deflate
AddOutputFilterByType DEFLATE text/html text/plain text/xml application/xml application/xhtml+xml text/javascript text/css
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

Lastly create a proxy.conf file in /etc/apache2/conf.d/…

<Proxy balancer://mongrel_cluster>
BalancerMember http://127.0.0.1:8000
AuthType Basic
AuthName “Supa Sekrit”
AuthUserFile /where/you/put/.htpasswd
Require valid-user
</Proxy>

One all that is done you can cd into the CruiseControl.rb directory and issue, ruby ./cruise start -p 8000 –daemon, and you should be ready to watch your tests fail like me.

Evolving Services on EC2

Friday, May 2nd, 2008

One of the great things about EC2 is that it is essentially a giant sandbox where you can take risks experimenting with architecture and services in a rapid and cost effective manner, something that you cannot do really well at co-lo or even on other VPS services.  In the past year we have experimented with plenty of different configurations: some found their way into production, others filed for future reference, and still some to be avoid all costs.

May 2007

When I came on board as a contractor we had only 2 servers inside EC2 and the database hosted at Go Daddy. The company had just migrated the Apache/Application server along with a Harvest server into EC2 but had opted to leave the database hosted at Go Daddy due to fears of data loss.  The only trouble with this scheme was the latency between the application and the db which made things so glacially slow that the site nearly unusable.

August 2007

After starting full-time we brought the database into the cloud and started looking into how we might implement a MySQL cluster in EC2.  The challenge was to get a backup routine that was unobtrusive yet fast and easy to transfer into S3. I never got LVM snapshots working to my comfort level so we relied instead on MySQLdump, which, all and all worked fine while the db was small. Data loss was still a big concern for us so we began experimenting in earnest with MySQL clusters.

November 2007

When the MySQL cluster idea didn’t pan out, the theory is that the small instances just didn’t have what it takes to cluster. So we went with plain old replication which has proven to be stable and reliable. The slaves serve both as fail-over units but also perform periodic backups freeing the master from that task.  Feeling more comfortable with database integrity we turned our attention to getting our application to scale, a challenge with resource hungry Rails.

January 2008

Breaking apart Apache and rails was a snap with mod_proxy and it allowed us to dedicate hardware to each.  With things running even better we started thinking about how we can flip this into a more through horizontal scale.

May 2008

So one year later and we have brought some horizontal scale to the site adding stability and failover to the application. As the site grows, though, we are back to the how we can best scale the database but at least we have a sandbox to play in so we can figure it out.

Of Monit and Mongrels, Quick Thoughts

Wednesday, March 5th, 2008

Monit has been serving us pretty well for the last 7 months or so in terms of keeping an eye on our mongrels and kicking them back in line if they act up. At the moment, we are running them in clusters of 3 with only 6 clusters in our current production set up and the monit restart all, for the most part, works fine when rolling them after a deploy. It is a completely different situation with 10 clusters which we are experimenting in a situation where Apache+mod_proxy sits on a separate server from the mongrels–it truly is wonderful to see Apache perform under load when it has all the resources it needs.

The problem seems to be with how resource hungry and ponderous Rails can be when firing up a mongrel, sucking up 25%+ user CPU and making the system gobble up another 10%+. This is enough so that when monit is trying to bring up or down 30 mongrels some of the pack gets left out, failing to either shutdown or start up. Now there is a monkey patch out there that addresses this very issue but I am a little wary in patching mongrel_cluster in our production environment as it might cause me headaches later with upgrades. So what is my solution?

Pressed for time it is a little kludgey and demonstrates some truly sloppy bash scripting but…it works.

#!/bin/bash
monit stop all -g pack_01
echo "Stopping 8100-02"
sleep 12s
monit stop all -g pack_02
echo "Stopping 8103-05"
sleep 12s
monit stop all -g pack_03
echo "Stopping 8106-08"
sleep 12s
monit stop all -g pack_04
echo "Stopping 8109-11"
sleep 12s
monit stop all -g pack_05
echo "Stopping 8112-14"
sleep 12s
monit stop all -g pack_06
echo "Stopping 8115-17"
sleep 12s
monit stop all -g pack_07
echo "Stopping 8118-20"
sleep 12s
monit stop all -g pack_08
echo "Stopping 8121-23"
sleep 12s
monit stop all -g pack_09
echo "Stopping 8124-26"
sleep 12s
monit stop all -g pack_10
echo "Stopping 8127-29"

So what I’ve done in the monitrc file is defined each mongrel as belonging to a group that reflects its cluster. Then I issue a stop to each group with a 12 second delay between each so that Rails and monit can navigate around each other with out either flipping out and forgetting to do their respective jobs. Starting is the same as above.

Does it work? Well, I have tried it under load several times (siege with a concurrency of 30 and the mongrels bloated up to our obese resting state) and it seems to work like a charm. I added the echos when our dev team expressed concern cap would time out while the script took its time to run all the way through.

File under: Ugly, Functional.