The Cult of Gary

I’m trying to do some simple load testing. I have a couple of web servers being “load balanced” with Round Robin DNS. That part is simple and is working fine.

I’m trying to simulate load using JMeter. JMeter has worked great when pointing at a single host so far. I’ve been able to spin up threads and hammer a server.

If I run JMeter from my Mac and point it at the RR DNS entry, it only hits one server for the TTL of the DNS entry. It will only change to a different server when the TTL expires.

This means all the requests go to one server for 20 minutes, then the other. I’ve made some changes to disable the Java level caching, which didn’t solve the problem. It appears to be an OS level thing. If I ping the entry I get the same IP address until I flush the cache with dscacheutil:

Macintosh-2:~ garyrichardson$ ping rr.dns.test
PING rr.dns.test (10.0.0.1): 56 data bytes
^C
--- rr.dns.test ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
Macintosh-2:~ garyrichardson$ ping rr.dns.test
PING rr.dns.test (10.0.0.1): 56 data bytes
^C
--- rr.dns.test ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
Macintosh-2:~ garyrichardson$ dscacheutil -flushcache
Macintosh-2:~ garyrichardson$ ping rr.dns.test
PING rr.dns.test (10.0.0.2): 56 data bytes
^C
--- rr.dns.test ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss


I’m using a free DNS service that doesn’t let me set the TTL lower than 1200 seconds.

Windows machines exhibit the same behavior, but you can apparently modify the registry to adjust the TTL. I’m not sure if this would actually fix my problem as I never tried it.

I tried two different Linux VM’s — a VMware instance on my laptop and an EC2 instance. They were both running CentOS 5. Both of these properly flipped DNS entries. But I found that when I run JMeter on a virtualized linux box, the threads hang. For example, if I set the thread count to 50, all 50 threads hit the server at once, then the performance seems to drop to about 1 req/second. 

I finally moved to a real linux box and was able to perform my tests. It’s across town and over a VPN, but it’ll get the job done.