EC2 and Ganglia
I’ve been playing around with Ganglia for the past couple of days, trying to make it to work with EC2. It was a bit of an adventure. There are two keys for running Ganglia on EC2: use unicast and set send_metadata_interval.
Amazon doesn’t support multicast on their network, so the default configs for Ganglia don’t work. You need to pick your head gmond server and set your udp_send_channel to something like:
udp_send_channel {
host = $headserver
port = 8649
ttl = 1
}
In your globals section, you also need to make sure that send_metadata_interval is set to something other than 0. From the mailing list:
Yep, you ran across the same dilemma I had when I wrote it :/ The problem is that in unicast mode, there is no requirement for any of the agents to be listening (ie. deaf = yes) since the individual nodes don’t need to do anything more than send their own current metric to the host node. So in this instance, attempting to send a request for metadata back to the node wouldn’t work. That is why I ended up just implementing the send_metadata_interval directive. The downside as you pointed out, is that more data is being passed needlessly on the wire however the amount of data should be less than with the older scheme. The reason why I say that is because under the old scheme, all meta data for any gmetric or modular metric was sent with every value packet rather than being sent independently. The intent was that the end_metadata_interval would be set to something on the order of minutes rather than seconds. This would mean that you might lose a few minutes of data if the host gmond were restarted, but the amount of useless metadata packets would be much less.
That last bit took some digging around to figure out. Without it, gmond knew about other hosts, but ignored the actual stats. Gmond wouldn’t record any info and graphs and data were missing.
