I finally reached that place where maintaining versions of host files just didn’t seem to cut it andfound myself mucking about with BIND9 to solve the problem of keeping track of all those internal IP addresses when I bring instances online and take them offline. This seemed to be the quickest and easiest way to handle fairly basic networking needs though it does introduce yet another point of failure so I’ll need to tackle primary and secondary DNS servers in the near future and maybe develop a doomsday host file backup but if the latter is invoked I’m sure that I have bigger problems on my hands.

Two basic tutorials helped me hack together the start of a solution, HowTo update DNS hostnames automatically for your Amazon EC2 instances and the ever useful Ubuntu wiki’s BIND9 Server Howto. Working back and forth between those two write-ups and a little manpage searching answered 95% of my questions so getting started isn’t as much of a headache as it might seem. Deploying it on a large scale? That is a whole other matter.

A caveat before we dive in, more to remind me when I have to revisit this some 18 months later. A couple of iterations in I thought it would be cute to set up the DNS zone for the full domain (*.awesome.com) but that seemingly added to my headaches as I tried to examine resources that did not have a DNS entry just yet. Since this is for internal use only we can pretty much set the FQDN to be whatever we want it to be, in this case pancake.man is our working domain.

Set things up…

apt-get install apt-get install bind9 dnsutils
sudo mkdir /etc/bind/zone/
sudo mkdir /some/where/other/than/etc/bind
sudo cp /etc/bind/db.local /etc/bind/zone/db.pancake.man
sudo cp /etc/bind/db.127 /etc/bind/zone/db.10
sudo chown -Rv bind:bind /etc/bind/zones/

EDIT: A couple of things occurred to me while I mulled this over. Keeping the zone files in /etc/bind while sounding like a neat idea doesn’t take into account the ephemeral nature of instances so I moved ours into an EBS volume. Additionally, I neglected to add that if your distro uses app armor you’ll have to grant rw permissions to the folder where you are keeping the db files. That can be done by editing /etc/apparmor.d/usr.sbin.named and adding the following:

/some/where/other/than/etc/bind/** rw,

The you can just reload apparmor and be on your happy way.

Generate a key pair….
This is really a CYA implementation but it should ensure that only the servers you want updating the LAN DNS can do it.

dnssec-keygen -a HMAC-MD5 -b 512 -n USER pancake.man

…and copy the secret from the public key as you’ll need it in the next section.

Set up a Zone…
Edit /etc/bind/named.conf.local and add the following:

key pancake.man. {
algorithm HMAC-MD5;
secret "Gzr11.....==";
};

The above block sets up the key that you created earlier and it will be used in the following block that defines the zone.

zone "pancake.man"
{
type master;
file "/etc/bind/zone/db.pancake.man";
allow-update { key pancake.man.; };
allow-query { any; };
};

Next is to work on the zone file so edit /etc/bind/zone/db.pancake.man and change the localhost/127.0.0.1 entries to fit your environment.

$TTL 604800
@ IN SOA ns.pancake.man. me.pancake.man. (
2 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS ns.pancake.man.
@ IN A 10.0.0.1 ; this is your local IP address

When you restart BIND9 this file will change a bit here’s what I see once it is running…

$ORIGIN .
$TTL 604800 ; 1 week
pancake.man IN SOA ns.pancake.man. me.pancake.man. (
3 ; serial
604800 ; refresh (1 week)
86400 ; retry (1 day)
2419200 ; expire (4 weeks)
604800 ; minimum (1 week)
)
NS ns.pancake.man.
A 10.208.41.206
$ORIGIN pancake.man.
$TTL 60 ; 1 minute

One thing that tripped me up for a little bit was the formation of the SOA line, pancake.man IN SOA ns.pancake.man. me.pancake.man.. I had been reading ns.pancake.man. me.pancake.man. as FQDN when in fact they are a FQDN and and email address where the ‘@’ has been replaced with a ‘.’; file this one under RIF: Reading Is Fundamental.

Next up is a reverse zone file which is pretty much the same process as what we just did, just edit /etc/bind/zone/db.10 and replace the localhost/127.0.0.1 values with your own.

;
; BIND reverse data file for local loopback interface
;
$TTL 604800
@ IN SOA ns.pancake.man. root.pancake.man. (
2 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS ns.
1.0.0 IN PTR ns.pancake.man.


Test your new DNS server
First things first, let’s test things out by inserting an entry using nsupdate (this piece is gratefully cribbed from Marius). Edit /etc/resolv.conf and put the IP address of your DNS server above the ec2 nameserver.

search compute-1.internal
nameserver 10.1.2.3
nameserver 172.1.2.3


Now lets punch an entry in using our shiny set of keys. I found making a script was easier for testing purposes, create a file called dnsupdate.sh and add the following:

#!/bin/bash
cat << EOF | /usr/bin/nsupdate -k Kpancake.man.+157+46088.private -v
server 127.0.0.1
zone pancake.man
update delete $1 A
update add $1 60 A $2
show
send
EOF

The private key is the one we made way back in the beginning of this exercise, your's might be about waffles. After making the file executable, just ./dnsupdate test.pancake.man 10.1.2.3. Ideally, you should get something like the following back:

Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 0
;; flags: ; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; ZONE SECTION:
;pancake.man. IN SOA


;; UPDATE SECTION:
test.pancake.man. 0 ANY A
test.pancake.man. 60 IN A 10.1.2.3


Implement your shiny new DNS server
With all that done now we and testing showing that everything works as expected we want to share it with the whole LAN but there's a potential snag. When you work up the next morning still heady from your success you noticed that when DHCP decided to check on its lease it wrote over /etc/resolv.conf with the stock EC2 values, showing absolutely no love for your handiwork. Not what we wanted to happen but there are a couple of steps we can take to make sure that your shiny new DNS persists (this is mainly geared toward Ubuntu so your mileage may vary). The AWS forums has a great writeup about this though I'm only implementing some what was discussed.

Since there is the chance of a DNS server changing, instance goes down or a new one is swapped in, I'm going the route of appending the DNS server to /etc/hosts at boot time and then adding the FQDN to /etc/dhcp3/dhclient.conf like so...

prepend domain-name-servers ns.pancake.man;

What that will do is punch in the IP address specified in /etc/hosts into /etc/resolv.conf when DHCP does it's little dance.

Loose ends...
While everything should be working decently on a small scale there are still plenty of things still left to do like setting up a secondary DNS server for failover, implementing EC2 internal zones for forwarding as described in the AWS post, and working out an elegant solution for updating clients about the DNS server. As for the latter, I'm still playing with scripts to fetch the local IP address of the DNS server, from using an empty security group as a tag to just keeping an updated list in S3 that is maintained by the DNS servers themselves. That, among the other things, will be a post for another day when I finally stumble on something that isn't so ham-fisted.