It was the damnest thing too. I was able to reboot it from the API and look at the console output. As far as I can tell, the network adapter wasn’t able to DHCP an IP address:


Welcome to CentOS release 5 (Final)
Press 'I' to enter interactive startup.
Setting clock : Mon Dec 22 20:06:28 EST 2008 [ OK ]
Starting udev: [ OK ]
Setting hostname localhost.localdomain: [ OK ]
No devices found
Setting up Logical Volume Management: No volume groups found
[ OK ]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
/dev/sda1: clean, 96119/1313280 files, 616984/2621440 blocks
[/sbin/fsck.ext3 (1) -- /mnt] fsck.ext3 -a /dev/sda2
/dev/sda2: clean, 5853/19546112 files, 892921/39092224 blocks
[ OK ]
Remounting root filesystem in read-write mode: [ OK ]
Mounting local filesystems: [ OK ]
Enabling local filesystem quotas: [ OK ]
Enabling /etc/fstab swaps: [ OK ]
INIT: Entering runlevel: 4
Entering non-interactive startup
Starting background readahead: [ OK ]
Checking for hardware changes [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0:
Determining IP information for eth0… failed.
[FAILED]
Starting auditd: [FAILED]
curl: (7) Failed to connect to 169.254.169.254: Network is unreachable
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Starting syslog-ng: [ OK ]
Starting irqbalance: [ OK ]
Starting system message bus: [ OK ]
Mounting other filesystems: [ OK ]
Starting sshd: [ OK ]
Starting cups: [ OK ]
Starting MySQL: [ OK ]
Starting postfix: [ OK ]
curl: (7) Failed to connect to 169.254.169.254: Network is unreachable
Starting httpd: Warning: DocumentRoot [/dev/null] does not exist
Warning: DocumentRoot [/dev/null] does not exist
Warning: DocumentRoot [/dev/null] does not exist
httpd: Could not reliably determine the server’s fully qualified domain name, using localhost.localdomain for ServerName
[ OK ]
Starting crond: [ OK ]
Starting process accounting: [ OK ]
Starting atd: [ OK ]
Starting jexec: Starting jexec services[ OK ]
Starting HAL daemon: [ OK ]
+ Updating ec2-ami-tools
curl: (6) Couldn’t resolve host ’s3.amazonaws.com’
c
CentOS release 5 (Final)
Kernel 2.6.16-xenU on an i686
localhost l

Luckily I had backups, so I booted a new instance and restored it it. It looks like I posted one article since my last backup. I reckon I’ll be able to get that back from a google cache.

It took me about 30 minutes to start the instance, restore the backup and test everything out. This time around, I set up an elastic IP. If this happens again, I won’t have to update DNS.

EDIT - this post was lost due to a server crash. I’m hoping to put it back in the same spot, but I’m not sure if wordpress will allow me to do that without ugliness.

I mostly like Confluence. I don’t like the {code} blocks. The highlighting and formatting sucks. There’s a limited number of languages supported and I don’t really code in any of them. The list from the documentation says:

Makes a preformatted block of code with syntax highlighting. All the optional parameters of {panel} macro are valid for {code} too. The default language is Java but you can specify JavaScript, ActionScript, XML, HTML and SQL too.

I’d really like a Confluence plugin that used the Google Code Prettify javascript code. These seem like obvious things to mate together, since you wouldn’t have to worry about implementing new languages.  Quite often, I write psuedo code in the wiki and the JS code does a decent job of figuring it out. I searched Google for confluence google-code-prettify, since someone must have built a plugin already. The results were not promising.

Dear Lazy Web, please see what you can do.

It took me a bit to get my perl test cases to work with it, but with the help of TAP::Harness::JUnit, I finally got it to work. It’s really easy too:

1. Install TAP::Harness::JUnit
2. Put the following script into you code repository. Put it somewhere that you can call it easily when doing a build.
#!/usr/bin/perl
use strict;
use warnings;
use TAP::Harness::JUnit;
my $outputfile = shift;
my $harness = TAP::Harness::JUnit->new({
xmlfile => $outputfile,
});
$harness->runtests(@ARGV);
3. Instead of calling make test for you perl modules, run:
   perl $SCRIPTFROMABOVE -Iblib/lib $OUTPUTFILE t/*.t

$OUTPUTFILE should be an xml file in whatever directory you have configured Bamboo to look in for test reports.

This should work with any CI server that can read JUnit xml output.

I once found an article about setting up a customer search engine that would allow you to permanently exclude sites. I’d have to remember to use my custom search engine every time, so I never tried it out.

My friend suggested that I use the Firefox plugin CustomizeGoogle. It allows you to filter sites in your results. I added /experts-exchange/ to the filter list and now Experts Exhange links are greyed out:

Hooray! No more bait and switch for me!

Amazon doesn’t support multicast on their network, so the default configs for Ganglia don’t work. You need to pick your head gmond server and set your udp_send_channel to something like:

udp_send_channel {
host = $headserver
port = 8649
ttl = 1
}

In your globals section, you also need to make sure that send_metadata_interval is set to something other than 0. From the mailing list:

Yep, you ran across the same dilemma I had when I wrote it :/ The problem is that in unicast mode, there is no requirement for any of the agents to be listening (ie. deaf = yes) since the individual nodes don’t need to do anything more than send their own current metric to the host node. So in this instance, attempting to send a request for metadata back to the node wouldn’t work. That is why I ended up just implementing the send_metadata_interval directive. The downside as you pointed out, is that more data is being passed needlessly on the wire however the amount of data should be less than with the older scheme. The reason why I say that is because under the old scheme, all meta data for any gmetric or modular metric was sent with every value packet rather than being sent independently. The intent was that the end_metadata_interval would be set to something on the order of minutes rather than seconds. This would mean that you might lose a few minutes of data if the host gmond were restarted, but the amount of useless metadata packets would be much less.

That last bit took some digging around to figure out. Without it, gmond knew about other hosts, but ignored the actual stats. Gmond wouldn’t record any info and graphs and data were missing.

That question came up on Stack Overflow, so I described my solution there.

I’ve just reached the load balancing part of my current project. LVS is a no go for EC2, since you need to be in the kernel and it depends on various natting and forwarding tricks to make it work. So I started my research and looked at a few different options.

I’m currently focusing on Nginx (pronounced Engine X). It’s a light weight web server that has some load balancing functionality built in. I haven’t had it in production, of course, but I really like the features and performance so far.

It handles spoon feeding slow clients. Nginx is a proxying load balancer. It buffers the transfer to the requester, which means the heavy weight apache processes can move on to the next request while nginx sends the data at the clients pace. This is something that LVS can’t do, since it’s a layer 4 app.

Since it’s proxying, nginx can manage backend servers going down. If nginx can’t talk to one backend server, it will move on to the next one. That’s neat, but it also means I don’t need to run another app to pull dead servers out of the load balancing pool.

Having the wrong IP in the logs is the only thing that sucks. The way around this is to use the X-Forwarded-For headers and tweaking your apps to read this instead of REMOTE_ADDR. And of course, if you care about the original X-Forwarded-For header (ie, AOL’s proxy), then you’ll need to mod the nginx config to pass this in a different header.

Apparently this is something that Xen does. I’ve found lots of posts referring to this, but none explaining why. Most people agree run level 4 is user defined. I suppose the Xen folks have defined a use for it!

My key requirements were to back up to S3 and to use encryption on the backups. I found a lot of apps/scripts/products out there that can do this. I had two “nice to have” requirements as well. I wanted to be able to automated the backups as much as possible. Also, I should be able to download and store the backups on my laptop as an additional copy.

I first tried Brackup. It worked OK and has some really nice features. For example, it can do synthetic backups, so you only need to do one full backup.  There are a few design issues with Brackup though.

For one, you can’t purge old backup files when using GPG encrypted backups. The catalog files are encrypt. The backup agent can’t decrypt the catalog files and figure out what files can be deleted. I also found my configuration files were huge. Directories and destinations get configuration blocks. Each directory you want to back up needs its configuration. You can have a single S3 destination, but if you want to be able to purge backups, you need to back up each directory to it’s own destination. I had 5-10 directories on each server to back up.

I gave up on Brackup and figured I’d write a shell script to tar and upload backups. Before I started coding, googled to see if anyone else had already written an app. I found s3napback. It’s a little more traditional; s3napback treats S3 like a tape device (using js3tream) and chains tar, gzip and gpg together to create full/incremental backups. 

It uses apache style configs that are nice and concise. S3napback is also much faster that Brackup. It doesn’t need to perform a new S3 transaction for each file. Plus, I can download the stream files and restore without any special tools on any machine that has tar and gpg. 

Data Usage - Your iPhone will be enabled for data usage. If you subscribe to a plan with no data included, data charges at a pay-per-use rate of 5¢/KB for data sent and/or received over the Rogers network will apply, unless you subscribe to a data plan, which we highly recommend. Roaming charges apply while using your iPhone outside Canada. U.S. data roaming on iPhone Packages is $3/MB. The standard international data roaming rate applies. Visit rogers.com/roaming for our roaming rates and destinations. Data usage is measured in KB rounded to the next full KB.

That means I can get an iPhone, keep my existing voice plan and turn the data off. Then, it’s wifi only! At least getting an iPhone won’t cost a fortune every month.

Now if they’d only offer a 1 or 2 year contract.