Stress testing with JMeter

Apache JMeter is an open source Java desktop application, built to verify functional behavior, perform load tests, and measure performance. Some of JMeter points of strength are mentioned here below:

  • It ships with an easy-to-use GUI;
  • It allows to set up multiple thread groups, different parallelism and request count, to ramp up the load;
  • It can use CSV files to generate semi-randomized requests;
  • It can execute parameterized tests;
  • It reports results in a simple table;
  • It uses assertions for checking test results;
  • It can be executed in batch.

In the picture below we have summarised the basic concepts of a JMeter test which we are going to build right afterwards.

../_images/jmeter11.png

JMeter basic concepts

Configuring a simple test

This section explains how to run performance tests using JMeter in order to evaluate the GeoServer performances when serving WMS requests. The performance test aims to stress the server and evaluates the response time and throughput with an increasing number of simulated users sending concurrent requests to the server.

Note

Ideally, to avoid adding extra load to the server JMeter should run on a different machine.

Warning

If you have performed the exercises in the security section, please go back to the layer and service security pages and open access to everybody, on all data and all services, before performing the exercises in this section

  1. From the training root, on the command line, run jmeter.bat (or jmeter.sh if you’re on Linux) to start JMeter:

    ../_images/jmeter2.png

    jMeter interface

  2. Add a new Thread Group with the mouse right click on Test Plan tree node:

    ../_images/jmeter3.png

    Adding a new Thread Group

  3. In the Thread Group panel set the number of thread for the test to 4 (this represents the number of simultaneous requests that are made to GeoServer) and the ramp-up period to 60. Also, chek Forever on the Loop Count field.

    ../_images/jmeter14.png

    Setting the Thread Group

  4. Add a new Loop Controller with the mouse right click on Thread Group tree node:

    ../_images/jmeter4.png

    Adding a new Loop Controller

  5. Right click on the Loop Controller tree node and add a new HTTP Request element:

    ../_images/jmeter5.png

    Adding a new HTTP Request

  6. In the HTTP Request enter the following basic configuration:

    Field Value
    Server Name or IP localhost
    Port Number 8083
    Path geoserver/ows
    ../_images/jmeter9.png

    HTTP Request host/port/path configuration

  7. From the training data dir root, open the data/jmeter_data/jmeter_request_params.txt, select and copy its contents in the clipboard, then click “Add from Clipboard” in the “HTTP request” panel to setup a sample GetMap request:

../_images/jmeter_request_params.png

HTTP parameters configuration

  1. Add the following listeners by right clicking on Test Plan tree node: View results Tree, Summary Report, Graph results

    ../_images/jmeter7.png

    Adding a Listeners

  2. Save this configuration into data/jmeter_data/ as jmeter_simple.jmx

At this point JMeter is configured to run a GeoServer performance test:

  1. Click on Run tool and select Start to startig the jMeter test.

    ../_images/jmeter13.png

    starting jMeter test

  2. Select View Results Tree to directly see the request informations produced and the request result:

    ../_images/jmeter15.png

    The View Results Tree panel

  3. Select Summary report to view the statistical informations about the requests:

    ../_images/jmeter19.png

    The Aggregate Graph panel

  4. Select Graph Results to analyze the technical trend of the requests:

    ../_images/jmeter17.png

    The Spline Visualizer panel

  5. Click on Run tool and select Stop to end the benchmarking session

Creating a Multiscale test

This chapter explains how to create a custom randomized Multiscale test with a set of multiple concurrent threads.

In the example before, we ran a test on a layer always requesting the same bounding box and image size. Now we should set up a more sophisticated test asking for different bounding boxes and resolutions, i.e. a multiscale test. Multiscale testing is useful because it stresses GeoServer requesting new data each time, mimicking real usage conditions and providing better testing for eventual SLD scale dependencies. With JMeter we can execute the test by passing a CSV file containing multiple randomized bounding boxes and image dimensions.

In the first paragraph we are going to describe how to generate a CSV file for randomized requests at different scales. In the second one is shown how to configure a new JMeter test with multiple simultaneous threads.

Create CSV file

  1. Only for Windows machines: open the file OSGeo4W.bat under %TRAINING_ROOT% folder inside the training home folder.

  2. Run:

    Linux:

    cd $TRAINING_ROOT/geoserver_data/data/boulder

gdalinfo srtm_boulder.tiff

    Windows:

    cd %TRAINING_ROOT%\geoserver_data\data\boulder

gdalinfo srtm_boulder.tiff

  3. The output of the command will be something like this:

    Driver: GTiff/GeoTIFF
Files: srtm_boulder.tiff
Size is 2520, 1800
Coordinate System is:
GEOGCS["WGS 84",
        DATUM["WGS_1984",
                SPHEROID["WGS 84",6378137,298.257223563,
                        AUTHORITY["EPSG","7030"]],
                AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4326"]]
Origin = (-105.700138888888890,40.300138888888888)
Pixel Size = (0.000277777777778,-0.000277777777778)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (-105.7001389,  40.3001389) (105d42' 0.50"W, 40d18' 0.50"N)
Lower Left  (-105.7001389,  39.8001389) (105d42' 0.50"W, 39d48' 0.50"N)
Upper Right (-105.0001389,  40.3001389) (105d 0' 0.50"W, 40d18' 0.50"N)
Lower Right (-105.0001389,  39.8001389) (105d 0' 0.50"W, 39d48' 0.50"N)
Center      (-105.3501389,  40.0501389) (105d21' 0.50"W, 40d 3' 0.50"N)
Band 1 Block=256x256 Type=Int16, ColorInterp=Gray
  Overviews: 1260x900, 630x450, 315x225, 158x113, 79x57, 40x29

  4. The information needed to create a multiscale CSV file are:

    Tile Size 256 x 256
    Pixel Size 0.000277777777778
    Bounding Box ((-105.7001389, -105.0001389), (39.8001389, 40.3001389))

  5. Run:

    Linux::
    cd $TRAINING_ROOT/data/jmeter_data

python ./wms_request.py -count 100 -region -105.7 39.8 -105.0 40.3 -minres 0.00028 -maxres 0.00224 -minsize 256 256 -maxsize 1024 1024 > multiscale.csv
    Windows::
    cd %TRAINING_ROOT%\data\jmeter_data

wms_request.py -count 100 -region -105.7 39.8 -105.0 40.3 -minres 0.00028 -maxres 0.00224 -minsize 256 256 -maxsize 1024 1024 > multiscale.csv

    wms_request.py is a python script which generates randomized requests at different bounding box and resolutions. The parameters are described in the following table:

    Parameter Description
    count Indicates the number of requests to generate
    region Indicates the maximum bounding box of each request
    minres/maxres Indicates the minimum and maximum value for the Pixel Size to request (Typically it should be at least the minimum resolution)
    minsize/maxsize Indicates the minimum and maximum dimensions of the requested image (Typically it should be at least as big as the tile size)

    The CSV file is structured following the rule $width;$height;$bbox.

    For example 290;444;-105.5904,39.910198,-105.48776,40.067338 indicates a request of size 290x444 and Bounding box [-105.5904,39.910198,-105.48776,40.067338].

    JMeter must be configured for parsing the CSV file correctly by using the CSV Data Set Config element.

Configure JMeter

  1. From the training root, on the command line, run jmeter.bat (or jmeter.sh if you’re on Linux) to start JMeter:

    ../_images/jmeter2.png

    jMeter interface

  2. Add three new items of type Thread Group called 1, 2, 4

  3. For each Thread Group set the Number of Thread (users) field equal to the Thread Group name, the Ramp-Up Period and Loop Count fields to 1.

    ../_images/jmeter18.png

    Setting the Thread Group (here: Thread Group 1)

  4. In the Test Plan section, check the Run Thread Groups consecutively checkbox

  5. Add a new Loop Controller to the first Thread Group object

  6. Right click on the Loop Controller tree node and add a new HTTP Request element with the same name of the Thread Group:

    ../_images/jmeter21.png

    Setting the HTTP Request

  7. In each HTTP Request add the following fields to the panel:

    Name Value Encode? Include Equals?
    bbox ${bbox} unchecked checked
    height ${height} unchecked checked
    width ${width} unchecked checked

    Which should look like in the picture

    ../_images/jmeter20.png

    HTTP Request panel configuration

  8. Uncheck the Follow Redirects and Use KeepAlive checkbox

  9. Right click on the Loop Controller tree node and add a new CSV Data Set Config element:

    ../_images/jmeter22.png

    Setting the CSV Data Set Config

  10. Configure the CSV Data Set Config as follows:

    Field name Value
    Filename ./multiscale.csv
    Variable names width,height,bbox
    Delimiter ;
    ../_images/jmeter23.png

    Configuring the CSV Data Set Config

  11. Right click the loop controller and choose “Copy”, then right click the other thread groups (one by one) and choose “paste” to get a full copy of the controller, HTTP sampler and CSV confiuration. Adjust each “HTTP request” node name to match the thread group name (1, 2 and 4)

  12. Configure the Loop Controller along the following schema:

    Thread Group
    • 1
    Loop Controller –> Loop Count
    • 100
    Thread Group
    • 2
    Loop Controller –> Loop Count
    • 50
    Thread Group
    • 4
    Loop Controller –> Loop Count
    • 50

  13. From the Test Plan tree node add an HTTP Request Defaults element (Config Element -> HTTP Request Defaults) and enter the following basic configuration:

    Field Value
    Server Name or IP localhost
    Port Number 8083
    Path geoserver/ows

    It should look like this:

    ../_images/jmeter9.png

    HTTP Default Request host/port/path configuration

  14. From the training data dir root, open the data/jmeter_data/jmeter_request_params_2.txt, select and copy its contents in the clipboard, then click Add from Clipboard in the HTTP request panel to set up a sample GetMap request:

  15. Add the following listeners by right clicking on Test Plan tree node: View results Tree and Summary Report

  16. Add the following assertions by right clicking on Test Plan tree node: Response Assertion

    ../_images/jmeter24.png

    Adding Assertions

    Note

    Using Assertions is helpful because it avoids to continuously do a visual check on the results.

  17. Configure the Response Assertion following this table:

    Field Value
    Apply to Main sample only
    Response field to test Response Headers
    Pattern Matching Rules Contains

    In the Pattern to test panel add:

    Content-Type: image/png

    The final result should look like in the picture:

    ../_images/jmeter25.png

    Configuring Response Assertion

  18. The overall test plan should look as follows. Save it in the jmeter_data directory as multiscale.jmx

../_images/multiscale.png

Saving the multiscale benchmark

At this point JMeter is configured to run a GeoServer performance test:

  1. Select the Test Plan tree node and select Run - Start from the top menu to start the JMeter test.

    ../_images/jmeter13.png

    starting JMeter test

  2. Select View Results Tree to directly see the request information produced and the requests results:

    ../_images/jmeter26.png

    The View Results Tree panel with a sample request

    ../_images/jmeter27.png

    Another request with different resolution and bounding box

  3. Select Summary Report to view the statistical information about the requests:

    ../_images/jmeter28.png

    Suymmary report panel