Tag Archive for ‘Amazon’ at elwoodicious

Tag Archive for 'Amazon'

Just a little self-congratulations…

Published by james on September 28, 2007 in Random. 0 Comments

So yesterday’s Amazon Web Services presentation went so much better than I could have ever imagined. We had a blast meeting so many new and interesting people who are dedicated to delivering some truly unique products. Two that stood out were AideRSS, which provides intelligent filtering to umanagable piles of RSS feeds and boasted the most amazing scaling of 100+ instances processing in parallel, and SeeMeWin.com, which has this quirky Japanese game show hook of watching people scratch lottery tickets. Part Jennicam, part “WTF seriously!?”, lead by a very engaging personality–Breck Yunits–SeeMeWin has the potential for localized traction among the 18-24 demographic.

The cap, though, was finding a link to this post, Amazon Web Services - EC2 - Wow!, in my email this morning.

There was also a lot of creativity leveraging EC2 to support production environment. Geezeo, located in Boston, have put everything on EC2: Front-end, app-servers, and database. Because MySQL replication and clustering is relatively easy, they could set up a small MySQL farm and then do frequent off-site backups to S3 (Amazon’s Simple Storage Service — you pay for that, but it’s not too expensive). Geezeo is sort of a mix of Quicken and Facebook. I’d been very leery of Geezeo because I don’t think I want my bank data up in the cloud. But after this presentation, I think they may have a good architecture for security; I might actually try them now. Which is saying a lot, because if you had told me their service was in EC2 before I saw their presentation, it would have actually increased my worry. They have SSL in the right places, and, apparently, private IPs running in EC2. Nice job.

Makes me feel real good about what we’ve built so far.

Here’s a copy of what we ran, Geezeo AWS Presentation and here’s what we built in EC2 (the part I’m crazy proud about!):

Geezeo Cluster

Speaking in Cambridge Today

Published by james on September 27, 2007 in Random. 4 Comments

Well, vacation is officially over.

This afternoon myself and our CTO are speaking at The Startup Project. We’ll be covering what the company is about and how we are leveraging Amazon Web Services to get things done.

It should go well and hopefully I can steel my nerves enough to make it through my five minutes of the presentation without too much stammering and sweating! This is such a seismic shift from my last job where I was largely invisible and the expectation was that I remain so; here in the past month I have attended a technology mixer where I spent time shaking hands and getting the word out (on the technical end) of what we are and how we are doing it to now speaking to a captive audience. I’ll be more than happy, though, to slip back behind the command line and not be wrapped up in counting how many times I said “um” in a minute.

Wish me luck!

Amazon S3, S3Sync, Backups, and You!

Published by james on June 22, 2007 in Programming. 4 Comments

With EC2 and S3 Amazon has made available some seriously powerful and flexible tools for server and file hosting. Ec2 allows you to roll whatever flavor OS you want and get it up and running–think of it as virtual dedicated hosting–which while being incredibly cool has one major downside: if your server instance fails you loose your data as it will revert to the most current instance when you get it back online. In other words, a hiccup on EC2 could turn into a blistering nightmare.

This is where S3 comes in handy for storing anything that might change after you create and instance and launch it such as databases, files uploaded or created, and even logs. One of the first projects I hopped into was dumping the MySQL databases, compressing them and tossing them up on S3, a tedious process that can be automated with cron and s3sync (a ruby based tool that approximates rsync, kinda sorta).

Daily cron job to create new bucket
cd /root/s3sync/ DAYNOW=$(date +%j) ruby s3cmd.rb createbucket WTF_db_$DAYNOW

Daily cron job to delete old bucket
cd /root/s3sync/ DAYTHEN=$(date +%j --date='2 days ago') ruby s3cmd.rb deletebucket WTF_db_$DAYTHEN

Hourly job to back up the database
# get into the directory cd /root/s3sync/

# set the environment export AWS_ACCESS_KEY_ID=XXXX export AWS_SECRET_ACCESS_KEY=XXXX/XXXX export SSL_CERT_DIR=/root/s3sync/certs

# set date variables DAYNOW=$(date +%j) TIMENOW=$(date +%H%M)
# dump database mysqldump WTF > /tmp/WTF-$DAYNOW-$TIMENOW.sql

# tar SQL dump tar -czvf /tmp/backups/WTF-$STAMPD-$STAMPT.tar.gz /tmp/WTF-$DAYNOW-$TIMENOW.sql

# copy tar to S3 ruby s3sync.rb -r --ssl /tmp/backups/ MahBukit:WTF_db_$DAYNOW

# clean up after yourself rm /tmp/*.sql rm /tmp/backups/*.gz

Now one of the frustrations I have with this set up is while I am dropping the buckets from 48 hours prior it isn’t actually deleting the files off S3, a bit of a pain in the ass that I need to find some sort of resolution for in the near future. If anyone has an answer to this problem I would love to hear it! For now, I’m cheating by using S3Fox to purge those pesky backups.