Effective load testing with Apache JMeter

Load testing is surely one of the most important activities that many developers ignore. I would include myself in that bracket; it is far too often something that gets bounced out of a busy schedule. However load testing, and its cousin stress testing, are absolutely essential when attempting to create a reliable application.

This blog post concentrates on load testing, which Wikipedia defines as:

Load testing is the process of putting demand on a system or device and measuring its response.”

This is subtly different to stress testing, which aims to test a system “beyond normal operational capacity, often to a breaking point”. Load testing is useful to ensure that your application meets the business requirements, for example “cope with x million page unique users per day with a response time of less than 1 second”.

This post covers:

Things to consider when load testing

It’s very easy to carry out useless load testing. I know this from experience. We were testing a system designed to serve 30 API requests per second, each response containing data on a user from a pool of roughly 100,000,000. When we carried out our load testing we designed a test using a pool of 1,000 unique user IDs. Spot the obvious problem! Testing with only 1,000 unique user IDs meant that we quickly reached a point where various layers of caching would be happily dealing with the job of fetching this limited set of data. This includes MySQL’s key cache plus the operating system file system cache.

To avoid making the same mistakes, make sure you consider the following points when load testing.

  1. Use indicative hardware
    When load testing to ensure that an application reaches minimum performance standards, you will need to test on hardware that mirrors your “live” setup as closely as possible. Sometimes this is easy, for example if you are running a live system consisting of three servers. If however you are running a 30 server cluster including a 100TB database, getting an accurate staging system can be hard work.
  2. Use indicative data
    If you have a production system with 100,000,000 rows of data, it’s a good idea not to carry out load testing on a staging system with only 300 rows. Similarly, if your application stores rows that are roughly 100KB each, the load testing system should be designed to replicate these conditions. The closer you can get to your actual data, the more useful the results.
  3. Think about system state
    Layers of caching can make load testing hard work. This includes application caching such as Memcache plus operating system caching and database caching. If your live environment workload can expect warmed caches then it is valid to leave these things warmed! However running a load test over and over again may get your application into an unnatural state of “preparedness”.

Using Apache JMeter

It may be tempting to write your own load test tools. Why not indeed? A few CURLs, a few processes; simple. My advice would be this: before you do this, try out JMeter. You can download JMeter from this page.

This example job reads in a list of user UUIDs from a text file and then makes an HTTP request for each user.

1. Launch JMeter

sh bin/jmeter.sh &

2. Configure job

  • Right click on Test Plan heading, click Add > Thread Group
  • Right click on Thread Group, click Add > CSV Data Set Config
  • Click on the newly added config element and enter a valid Filename – this should contain one user UUID per line
  • Enter userId under Variable Names (comma-delimited)
  • Click on the Thread Group, configure the Number of Threads (say 5), the Ramp-Up Period (say 180) and then the test Duration (say 600). This tells JMeter to launch up to 5 threads, ramping these up over 2 minutes, with the test running for a total of 10 minutes.

Thread group settings

  • Right click on Thread Group, click Add > Sampler > HTTP Request
  • Click on the newly added sampler element and enter a valid Server Name or IP
  • You should probably add in some Timeouts; depending on your desired performance under load
  • Enter a Path, including the previously defined variable as follows: /1/user/${userId}/data.json

Screen shot 2010-09-23 at 14.41.36

3. Add a listener

This was a stage that took me a while to figure out! You must have at least one listener, otherwise you cannot view any results from the job. You can add listeners to the Thread Group in the same way you added samplers and config elements. The Summary Report is very useful, as is View Results Tree (especially during debugging).

4. Run the job!

It’s usually a good idea to Clear All (from the Run menu) before you fire off the job.

5. Profit

Epic Fail!

Verifying the details of a response

By default JMeter will work with HTTP response codes – which is handy. So 404 errors and 500 errors etc… will all be dealt with. It’s often useful to look at the actual content of the response to decide on success. This can help catch things like PHP Fatal Errors (where you may have a 200 OK response, but a completely blank document). This is where the Response Assertion comes in.

Right click on any sampler and then Add > Assertion > Response Assertion. You can then define patterns to test against; these can use regular expressions. One other point to note is that if you wish to have HTTP 500 or 400 errors ignored (treated as success) then you should tick the Ignore Status tick box.

Extracting data from a response

Hitting URLs with known UUIDs is a handy tool, however it’s also useful to call a URL to add a user, extract the newly added user UUID and then use this in any subsequent requests. Fortunately this is also very easy to achieve with JMeter, using the Regular Expression Extractor.

Right click on any sampler and then Add > Post Processors > Regular Expression Extractor. The following settings extract data from a JSONP response which includes a userId definition; this is then assigned to the JMeter variable userId.

  • Reference Name: userId
  • Regular Expression: var userId = ‘([a-f0-9-]+)’;
  • Template: $1$
  • Match No. (0 for Random): 1
  • Default Value: null

Configuring variable extraction

Defining load levels using a constant throughput timer

When load testing I find it useful to run a number of tests at different levels of load – for a very specifically defined number of requests per second. To achieve this level of control, the Constant Throughput Timer element is very handy. This timer allows you to specific a target throughput in samples per minute. JMeter will then throttle back requests, if needed, to attempt to achieve this rate. The only thing you may need to do is ensure you have enough threads allowed to meet the desired throughput. Remember that the number of threads can be defined within the Thread Group settings.

Timer settings

Enjoy your load testing!

Tags: , , ,

6 Responses to “Effective load testing with Apache JMeter”

  1. Raul says:

    pleas provide more information regarding detecting faults in the DB response times. I added some Assertions like Response and Duration.
    But the problem is that in Response assertion if i add a patter to test . for example in one jdbc request i am selecting all fields of a table and i added the value of one of the records in the Response assertion. but how do i keep this assertion only applicable for one jdbc request and not for others. because when i run then it fails for all.

  2. Sri says:

    Excellent Article! Thanks for sharing!! Can you also please tell me how to use Jmeter to verify if the content is served from cache? We have been sticking to some browser extensions for this purpose. But they were not very accurate though.I am thinking, may be I am not using the appropriate browser extensions. Please share if you have used any browser extensions used for this purpose as well.

    Thank in advance!

  3. Dave says:

    I’m assuming you have control over the website you’re testing – therefore why not just turn of caching if that’s what you want? Or alternatively you might be able to make use of HTTP headers and inspect the contents.

  4. Dave says:

    I’m not entirely sure, however you have full control over the timeouts per-sample so that might help you out.

  5. Dzmitry Kashlach says:

    Thank you for article.
    Indeed, it is better to replicate production conditions, when we are configuring test. And here we have different solutions:
    – using separate test-lab.
    – using cloud services, such as JMeter testing cloud.

    Both of them have advantages and disadvantages.
    For example, when you use cloud, you cannot isolate yourself from Internet space. Therefor, you may have redundant traffic, which can be mixed with traffic of performance test. And this will affect statistics, if you want to test it “as it is”.
    From the other hand, when your system is already in production, you can get results which will mirror work in real Internet. And that’s, IMHO, a plus.

  6. SAF says:

    Hi Dave,

    Thanks for the article. However, can you please elaborate on how to pick up the parameters from the CSV file? For example, for HTTP POST request, how do I tell it to pick up the UUIDs from the CSV file? Can you please put up the screenshot of the CSV Data Set Config?

    Thanks a lot!

Leave a Reply