Posts Tagged ‘EBS’

Re-Sizing EBS Volumes for Fun and Profit on EC2

Thursday, September 17th, 2009

This is one you can file under easy and obvious but since I have a memory like a sieve I am going to write about it. There are times when I set up EBS volumes and think to myself, “I’ll never need anymore than nGB, ever!”, only to find out some months down the road my estimates were woefully short for the growth trend. Turns out resizing is pretty easy and can be accomplished in a matter of minutes. For this example, I’m going to resize a single EBS volume using XFS and where a MySQL database hangs out doing its thing. I am going to attach it to a separate block device than the original volume so that we can quickly revert back if we find the apocalypse happening ahead of schedule.

  1. Put up a maintenance page or just halt activity to the DB so that your life is slightly more elegant.
  2. Stop MySQL
  3. Make a snapshot of the original EBS volume
  4. Create a new volume from that snapshot specifying the size you want and make sure that it is in the same availability zone as the instance you want to attach it to
  5. Attach the new EBS volume to your instance on a different block device (if the original is on /dev/sdh then attach the new one to /dev/sdi)
  6. Edit /etc/fstab to reflect the new changes:

    #/dev/sdh /vol xfs noatime 0 0
    /dev/sdi /vol xfs noatime 0 0

  7. Mount the drive
  8. Resize it with “xfs_growfs -d /vol”
  9. Start MySQL and let it run its checks
  10. Take down the maintenance pages and sit back with a sense of smug self-satisfaction

Like I said, super easy. However, this is for a single EBS volume I haven’t really played around with resizing or generally manipulating RAID sets so that is a post for another day.

Tags: , , ,
Posted in Sysadmin | 1 Comment »

Save your database (and your bacon) with Elastic Block Store and mysqlcheck

Monday, March 16th, 2009

Here’s the situation: early this afternoon I get a panicked IM from a client that they dropped a table on the production db but that they have a CVS copy that they want to load.  Sounds easy, right?  Should have been but the CSV file had some oddities where lines were terminated by \n but those also existed in some of the fields.  Now, I am by no means a MySQL guru and while there might be a solution to issue a LOAD DATA INFILE statement that accounts for it the process of working around that would keep the site down longer than necessary.

When we setup the database on EC2 we made a conscious decision to move the database into an Elastic Block Store (EBS) so that we could take regular snapshots of the volume as well as enjoy the durability that they offer.  The approach that we took to recovering the table was to re-purpose one of the QA instances as a recovery point, create an EBS volume from the latest snapshot, point MySQL at it, dump the table with –complete-insert, source it in, call it a day.

Things were working swimmingly up until the point where I needed to dump the table, ERROR 1033 (HY000): Incorrect information in file, was the last thing I wanted to read.  The table is InnoDB and it is possible that it was corrupted when I started the server back up with some missing variables in the my.cnf file–lesson here is remember to breathe, work quickly but methodically, and double check your work.  So here I sat with a seemingly good copy of the database but a mangled table.

Just a handful of keystrokes saved my ass: mysqlcheck mydb mytable.

If we had been doing just whole database dumps with mysqldump this process would have been frustrated by trying to chop up a 12GB file into the section that we needed (yes, it makes more sense to dump each table seperately but remember, I’m no guru).  In the end, having a volume that we could mount and access withing minutes was the biggest reason we were able to get the production site back online as fast as we did and for future reference I’ll check my config files more closely before I turn on services.

***Note: this really only applies if your database runs in EC2 ;-)

Tags: , , ,
Posted in Sysadmin | Comments Off