Which gets the measurements right, JMeter or Apache ab?

Question

I started writing some basic tests in JMeter and was surprised that the measurements are so different from those from Apache ab.

I have a gigabit LAN connecting an Inte

Answer 1

In your setup, JMeter is saturating itself faster than it can saturate your web server.

You are running a very optimized C web server on superior hardware and bench-marking it with a relatively heavy Java app on lesser hardware. Optimized C machine code will (probably) always be faster than Java bytecode. JMeter is not able to keep up with Nginx and is therefore giving you strange results as it hits hardware limitations. Java does lots of nice things in the background that manage hardware resources, but also create unpredictable behavior at extreme resource usage. ApacheBench, on the other hand, is a light-enough C program that it can saturate the server and can produce consistent results because it has excess capacity after saturating your web server.

JMeter is great for bench-marking heavy-er dynamic applications that need some time to process requests. All the extra data it provides helps with web apps like that. When you're dealing with static file serving (just about the fastest thing a web server can do) on highly optimized web servers, you need a tool fast enough to keep up.