Archive for the ‘Linux’ Category

Getting your tunnel on with EC2 and OpenVPN (or wrap it before you ride it)

Tuesday, November 16th, 2010

I worry about data. I worry about whether it’ll be lost, compromised, or corrupted. I worry whether or not I am doing enough: did I check the backups, are they “good”, did any fail to run, do I have a “plan”, will it work? It is easier for me to get a handle on my fretting with regards to the server I tend but what about those assets that get to see the wider world and networks other than my own?

As a company, we are traveling more and it isn’t unusual for people to work from airports, coffee shops, clients, taxi cabs, or bars. Its just how business is getting done. But what about that data I worry about so much, both company and personal? Things like Firesheep intrigue me at the same time as leave me with the feeling that I need to do more for the people on the road.

The most obvious solution is to implement a VPN but which one, how, and moreover could we run it in our EC2 stack. I experimented with PPTP and loved the ease and simplicity of setting it up and it worked great with the built in OS X VPN client. However, once I left the free love environment that is my home network it was all but useless as I encountered plenty of public WiFi that disallowed TCP/1723. Next up was Openswan’s IPSec implementation, a complex beast full of options and configurations. It enjoyed wide support on the public networks I tried but, sadly, I seemingly could not get the built in OS X VPN client to consistently play nice. Last up was OpenVPN.

OpenVPN is easy to setup, configure, and roll out. We use it on our production servers as part of the CohesiveFT VPN-Cubed product that allows us to easily throw up IPSec tunnels with clients and hook it back stack so we know it is rock solid. The only drawback that I encountered with OpenVPN is that neither the iPad nor the iPhone support it so this does limit the devices that we can secure at the moment but life is about compromise.

Eric Hammond wrote an excellent article as a proof-of-concept for running OpenVPN over TCP/80 on EC2. Per usual he made it dead simple and easy to implement and with the slightest bit of finessing it can be adapted for rolling out to small teams.

For this I chose a micro instance running Ubuntu 10.04 for both the typical miserly and familiarity reasons. Prepping I ran with Hammond’s opening tasks:

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install openvpn -y
sudo modprobe iptable_nat
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
sudo iptables-save > /some/place/you/can/recall
sudo iptables -t nat -A POSTROUTING -s 10.4.0.1/2 -o eth0 -j MASQUERADE

Once that inital prep work was out of the way I hoped over to the official Ubuntu OpenVPN documentation where we can set up OpenVPN to use asymmetrical keys instead.

But first we need to make a couple of tweaks to the server config (/etc/openvpn/server.conf). Look for the following and adjust them:

port 80
proto tcp
server 10.4.0.0 255.255.255.0

The port and protocol is so that our tunnel will run over TCP/80 and this, in theory, let you toss up your tunnel on nearly any network that has access to the Internet. That said, I’ve tested this on my free love network, a tighter administered open WiFi where my daughter goes to school (this is the one that disallows TCP/1723), and my T-Mobile data plan and while it worked on all three your mileage may vary.

The next step is to create a CA for OpenVPN and generate keys for clients and the server. Rather than do that work in /etc/openvpn/ as suggested in the documentation I opted to perform all that work in another directory so we can easily manage it with tools like git or puppet.

sudo mkdir /mnt/easy-rsa/
sudo cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/* /mnt/easy-rsa/
sudo chown -R $USER /mnt/easy-rsa/

Adjust the variables in /mnt/easy-rsa/vars to best fit how you want things to look:

export KEY_COUNTRY="US"
export KEY_PROVINCE="CT"
export KEY_CITY="Hartford"
export KEY_ORG="Awesome Pancake, LLP"
export KEY_EMAIL="muffin.man@awepancake.com"

Then, because the devs and maintainers are awesome, just run some handy scripts and you’ll be all set:

cd /mnt/easy-rsa/
source vars
./clean-all
./build-dh
./pkitool --initca
./pkitool --server server
cd keys
openvpn --genkey --secret ta.key
sudo cp server.crt server.key ca.crt dh1024.pem ta.key /etc/openvpn/

Now we just need to generate some client certs:

cd /mnt/easy-rsa/
source vars
./pkitool jimmy

Replace “jimmy” with whatever you want to name your client, I recommend easy to remember nicknames that will cause you to snicker when you use them.

Now for the clients, it all depends on the what you’ll be using to connect to the server with. For testing I tried Tunnelblick and while it’s FLOSS I found it to be a little harder to configure than Viscosity, which is an affordable commercial client. The additional benefit to Viscosity is that it will allow you create easy to distribute connection bundles that include the CA cert, client key, and client cert that you can distribute to your clients for importing.

Setting up Viscosity is about as easy as it gets, you’ll need the CA.crt, and the host key and cert that you created above. Since pictures are a more valuable currency:

Once that is all set, connect to your tunnel and marvel at your secure connection. To check how things look on the client you can check ifconfig to ensure that the tunnel was added:

Also, do a quick nslookup to ensure that DNS is going to EC2 rather than to the local network:

Viscosity as a nifty detail page for visualizing the traffic over the tunnel:

On the server you can use tcpdump to watch the traffic and make sure that it is flowing like you want:

sudo tcpdump -i tun0 | grep "ip-10-4-0-6.ec2.internal"

Where 10-4-0-6 is the same as the ip address given by the tunnel.

That’s pretty much it, very easy thanks to Eric Hammond and the Ubuntu documentation team. From here you could set up connections in Viscosity for each of your users, export a zipped connection and distribute it (securely, of course!).

The next big test will be setting up all our road warriors and asking them to try it while they are traveling and report back with concerns, problems, or suggestions because in the end I want it to be easy as possible as they’ll be more inclined to use it.

I hate Ubuntu 9.10′s GDM theme so much I tweaked it slacker style.

Thursday, October 29th, 2009

Quick, sloppy, and gratefully cribbed from rfurgy’s comment on a post about hacking the new GDM, which incidentally his comment was the only method that worked out of all the various ways being discussed. Maybe it’s me, maybe it’s not but this method, while kind of silly, actually does make a change though ymmv.

The fugly wallpaper is kept in /usr/share/images/xsplash as bg_[screen resolution].jpg along with the logos and something called a throbber. Make backups of these and tuck them away in a safe place in case you want to revert back.

  1. Make backups of all the files in /usr/share/images/xsplash just because it is the responsible thing to do
  2. Find an image that you want to use for your login background
  3. Crop it to the resolution of your screen
  4. Generate resized versions for the six sizes contained in the folder
  5. Copy your new images into /usr/share/images/xsplash
  6. If you are using a light or white colored back ground you might want to invert the color in the logo images
  7. Log out or reboot
  8. Enjoy a little less ugly in your life

I’d take a screenshot but gdmflexserver –xnest tells me it hasn’t been implemented and I’m too lazy to look for another way. Shaky and dimly lit iPhone pictures are worth a handful of words.

Cheeseball Screenshot

Quick Thoughts on Ubuntu 9.10 RC

Saturday, October 24th, 2009

I made the jump to the release client last night, well so late that I woke up to finish the upgrade in the middle of the night because it took so long to pull down all the files. Here’s a brain dump of things I’ve noticed but bear in mind that fixes and changes might be pushed before official launch in a weeks time.

Lowlights

Hate the Simple Greeter for GDM and loathe the broken functionality of GConf-Editor. If you want to dump the user list at the login page issue the following on the cli => sudo -u gdm gconftool-2 –type bool –set /apps/gdm/simple-greeter/disable_user_list ‘true’ (Thanks to Gramps at Ubuntu Forums). Bottom line, I hate user lists for login and it is a personal preference. Moreover, it chaps my ass when the supposed tools for adjusting behaviors of apps fail to work. Adding salt to that chapped ass is the UX disaster that the simple-greeter becomes after you disable the user list, tapping the enter key to type your own name is senseless and counter-intuitive.  If the user list is disabled present the username input box.

The disk check on boot feels like a UX regression as it a tiny terminal window that continually refreshes the overall state of the scan, the UUID and percent complete, in a scrolling list. If the the information isn’t changing beyond the percent why not just so that clocking up? The disk check dialog from 9.04 was much more polished and it is possible that 9.10 will implement something similar so I’ll reserve judgment.

Sound events has taken on an odd crackle, like the discharge of static, when my laptop is muted. No idea what that is all about but I have been going back through all the preferences and been disabling sound events since I am more a visual person for notifications.

Highlights

Gwibber 2.0.0 Gwibber 2.0.0 is double-plus good. Polished, clean, and solid. Best experience so far was that on launch it recognized that Facebook was in an incomplete configuration state and walked me through finishing it. Well done, Team Gwibber!

Shutdown does feel faster over earlier releases but boot is a mixed bag in that it feels like you get to the login screen at a decent time but strapping the desktop environment after login feels slow; could be a Compiz issue but I am not sure.  I’m waiting to try suspend and hibernation to see if those improved any as my experience has been that both were notoriously slow.

Empathy, in my 60 second whirlwind setup, seems to be much more polished then when I last used it earlier this year.  I was able to set up all my accounts including my work chat via Google Apps with absolutely no issues.  The integration with the desktop is nice though I’m still trying to get my head around how to call up the chat list without invoking Gnome-Do. Also, it’s implementation of conversation logging is very nice.

Ubuntu One is much less bitchy about a lagging Internet connection and now will re-connect without forcing me to re-authorize my laptop each and every time. +1 for basic functionality.

The usual raft of “essential” apps (Firefox, Tasque, Gnome-Do, ec2 tools) have been working without a hitch which is nice because as far as I can tell no updates occurred for those programs (well, alright, I was already on FF 3.5 and I fetch my own ec2 tools).

Summary

So far so good but like any new release it is a little bit of a mixed bag. I’m hopeful that somethings will improve after the launch, like the GDM, but also very pleased that doing an early upgrade didn’t put me in the position of firefighting. All that said, this is a very solid release and after the official launch I will have no problems recommending it to Management or my mother-in-law.

EC2, BIND9, DNS, Pancakes and You!

Wednesday, August 12th, 2009

I finally reached that place where maintaining versions of host files just didn’t seem to cut it andfound myself mucking about with BIND9 to solve the problem of keeping track of all those internal IP addresses when I bring instances online and take them offline. This seemed to be the quickest and easiest way to handle fairly basic networking needs though it does introduce yet another point of failure so I’ll need to tackle primary and secondary DNS servers in the near future and maybe develop a doomsday host file backup but if the latter is invoked I’m sure that I have bigger problems on my hands.

Two basic tutorials helped me hack together the start of a solution, HowTo update DNS hostnames automatically for your Amazon EC2 instances and the ever useful Ubuntu wiki’s BIND9 Server Howto. Working back and forth between those two write-ups and a little manpage searching answered 95% of my questions so getting started isn’t as much of a headache as it might seem. Deploying it on a large scale? That is a whole other matter.

A caveat before we dive in, more to remind me when I have to revisit this some 18 months later. A couple of iterations in I thought it would be cute to set up the DNS zone for the full domain (*.awesome.com) but that seemingly added to my headaches as I tried to examine resources that did not have a DNS entry just yet. Since this is for internal use only we can pretty much set the FQDN to be whatever we want it to be, in this case pancake.man is our working domain.

Set things up…

apt-get install apt-get install bind9 dnsutils
sudo mkdir /etc/bind/zone/
sudo mkdir /some/where/other/than/etc/bind
sudo cp /etc/bind/db.local /etc/bind/zone/db.pancake.man
sudo cp /etc/bind/db.127 /etc/bind/zone/db.10
sudo chown -Rv bind:bind /etc/bind/zones/

EDIT: A couple of things occurred to me while I mulled this over. Keeping the zone files in /etc/bind while sounding like a neat idea doesn’t take into account the ephemeral nature of instances so I moved ours into an EBS volume. Additionally, I neglected to add that if your distro uses app armor you’ll have to grant rw permissions to the folder where you are keeping the db files. That can be done by editing /etc/apparmor.d/usr.sbin.named and adding the following:

/some/where/other/than/etc/bind/** rw,

The you can just reload apparmor and be on your happy way.

Generate a key pair….
This is really a CYA implementation but it should ensure that only the servers you want updating the LAN DNS can do it.

dnssec-keygen -a HMAC-MD5 -b 512 -n USER pancake.man

…and copy the secret from the public key as you’ll need it in the next section.

Set up a Zone…
Edit /etc/bind/named.conf.local and add the following:

key pancake.man. {
algorithm HMAC-MD5;
secret "Gzr11.....==";
};

The above block sets up the key that you created earlier and it will be used in the following block that defines the zone.

zone "pancake.man"
{
type master;
file "/etc/bind/zone/db.pancake.man";
allow-update { key pancake.man.; };
allow-query { any; };
};

Next is to work on the zone file so edit /etc/bind/zone/db.pancake.man and change the localhost/127.0.0.1 entries to fit your environment.

$TTL 604800
@ IN SOA ns.pancake.man. me.pancake.man. (
2 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS ns.pancake.man.
@ IN A 10.0.0.1 ; this is your local IP address

When you restart BIND9 this file will change a bit here’s what I see once it is running…

$ORIGIN .
$TTL 604800 ; 1 week
pancake.man IN SOA ns.pancake.man. me.pancake.man. (
3 ; serial
604800 ; refresh (1 week)
86400 ; retry (1 day)
2419200 ; expire (4 weeks)
604800 ; minimum (1 week)
)
NS ns.pancake.man.
A 10.208.41.206
$ORIGIN pancake.man.
$TTL 60 ; 1 minute

One thing that tripped me up for a little bit was the formation of the SOA line, pancake.man IN SOA ns.pancake.man. me.pancake.man.. I had been reading ns.pancake.man. me.pancake.man. as FQDN when in fact they are a FQDN and and email address where the ‘@’ has been replaced with a ‘.’; file this one under RIF: Reading Is Fundamental.

Next up is a reverse zone file which is pretty much the same process as what we just did, just edit /etc/bind/zone/db.10 and replace the localhost/127.0.0.1 values with your own.

;
; BIND reverse data file for local loopback interface
;
$TTL 604800
@ IN SOA ns.pancake.man. root.pancake.man. (
2 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS ns.
1.0.0 IN PTR ns.pancake.man.

Test your new DNS server
First things first, let’s test things out by inserting an entry using nsupdate (this piece is gratefully cribbed from Marius). Edit /etc/resolv.conf and put the IP address of your DNS server above the ec2 nameserver.

search compute-1.internal
nameserver 10.1.2.3
nameserver 172.1.2.3

Now lets punch an entry in using our shiny set of keys. I found making a script was easier for testing purposes, create a file called dnsupdate.sh and add the following:

#!/bin/bash
cat << EOF | /usr/bin/nsupdate -k Kpancake.man.+157+46088.private -v
server 127.0.0.1
zone pancake.man
update delete $1 A
update add $1 60 A $2
show
send
EOF

The private key is the one we made way back in the beginning of this exercise, your's might be about waffles. After making the file executable, just ./dnsupdate test.pancake.man 10.1.2.3. Ideally, you should get something like the following back:

Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 0
;; flags: ; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; ZONE SECTION:
;pancake.man. IN SOA


;; UPDATE SECTION:
test.pancake.man. 0 ANY A
test.pancake.man. 60 IN A 10.1.2.3

Implement your shiny new DNS server
With all that done now we and testing showing that everything works as expected we want to share it with the whole LAN but there's a potential snag. When you work up the next morning still heady from your success you noticed that when DHCP decided to check on its lease it wrote over /etc/resolv.conf with the stock EC2 values, showing absolutely no love for your handiwork. Not what we wanted to happen but there are a couple of steps we can take to make sure that your shiny new DNS persists (this is mainly geared toward Ubuntu so your mileage may vary). The AWS forums has a great writeup about this though I'm only implementing some what was discussed.

Since there is the chance of a DNS server changing, instance goes down or a new one is swapped in, I'm going the route of appending the DNS server to /etc/hosts at boot time and then adding the FQDN to /etc/dhcp3/dhclient.conf like so...

prepend domain-name-servers ns.pancake.man;

What that will do is punch in the IP address specified in /etc/hosts into /etc/resolv.conf when DHCP does it's little dance.

Loose ends...
While everything should be working decently on a small scale there are still plenty of things still left to do like setting up a secondary DNS server for failover, implementing EC2 internal zones for forwarding as described in the AWS post, and working out an elegant solution for updating clients about the DNS server. As for the latter, I'm still playing with scripts to fetch the local IP address of the DNS server, from using an empty security group as a tag to just keeping an updated list in S3 that is maintained by the DNS servers themselves. That, among the other things, will be a post for another day when I finally stumble on something that isn't so ham-fisted.

Ubuntu 8.10

Thursday, October 30th, 2008

Jumped on the RC just before it went official late this afternoon (EST) so that I could beat the crowds.  One of the faster upgrades that I have experienced with the package downloads taking around 20 minutes on my 6MB DSL and the install and clean up taking a little under an hour.  So easy it was kind of boring.

Highlights are the revamped network manager which now has support for mobile connections which will make it easier for me to poach one of the Verizon data cards from work should the need ever arise. The new Bluetooth wizard also appears to have upped the functionality and UI quotient so I’m looking forward to pulling out my headphones and going wireless again.

Downsides are that the shutdown button has been subsumed to logout/fastuser switching something I nearly never do but could understand the logic for a desktop user but, seriously, who uses a desktop these days anyhow? Unrelated to the actual upgrade but still nagging issues are the Glipper crash on launch and the fact that my webcam still remains unsupported (05e3:f192 Genesys Logic, Inc.)–I’ve got to complain about something, right?

Overall, I’m happy with how things turned out.  Thumbs up to the Ubuntu team for all their hard work.  Thanks!

Building Google Chrome on Linux

Wednesday, September 3rd, 2008

Why not play along? It is that exciting. Like a Gentoo install fest.

/edit

A better part of the afternoon flushed down the drain for two executables: base_unittests and net_unittests.  Passed with flying colors but neither is a browser.

rm -rf /chrome