With EC2 and S3 Amazon has made available some seriously powerful and flexible tools for server and file hosting. Ec2 allows you to roll whatever flavor OS you want and get it up and running–think of it as virtual dedicated hosting–which while being incredibly cool has one major downside: if your server instance fails you loose your data as it will revert to the most current instance when you get it back online. In other words, a hiccup on EC2 could turn into a blistering nightmare.
This is where S3 comes in handy for storing anything that might change after you create and instance and launch it such as databases, files uploaded or created, and even logs. One of the first projects I hopped into was dumping the MySQL databases, compressing them and tossing them up on S3, a tedious process that can be automated with cron and s3sync (a ruby based tool that approximates rsync, kinda sorta).
Daily cron job to create new bucket
cd /root/s3sync/
DAYNOW=$(date +%j)
ruby s3cmd.rb createbucket WTF_db_$DAYNOW
Daily cron job to delete old bucket
cd /root/s3sync/
DAYTHEN=$(date +%j --date='2 days ago')
ruby s3cmd.rb deletebucket WTF_db_$DAYTHEN
Hourly job to back up the database
# get into the directory
cd /root/s3sync/
# set the environment
export AWS_ACCESS_KEY_ID=XXXX
export AWS_SECRET_ACCESS_KEY=XXXX/XXXX
export SSL_CERT_DIR=/root/s3sync/certs
# set date variables
DAYNOW=$(date +%j)
TIMENOW=$(date +%H%M)
# dump database
mysqldump WTF > /tmp/WTF-$DAYNOW-$TIMENOW.sql
# tar SQL dump
tar -czvf /tmp/backups/WTF-$STAMPD-$STAMPT.tar.gz /tmp/WTF-$DAYNOW-$TIMENOW.sql
# copy tar to S3
ruby s3sync.rb -r --ssl /tmp/backups/ MahBukit:WTF_db_$DAYNOW
# clean up after yourself
rm /tmp/*.sql
rm /tmp/backups/*.gz
Now one of the frustrations I have with this set up is while I am dropping the buckets from 48 hours prior it isn’t actually deleting the files off S3, a bit of a pain in the ass that I need to find some sort of resolution for in the near future. If anyone has an answer to this problem I would love to hear it! For now, I’m cheating by using S3Fox to purge those pesky backups.
