JMeter quick start guide (part 4)

--------------------------
4. Understanding results
--------------------------

1. As a history of each run, I recorded information about each run in a file called "description of run.txt" in which I'd describe things like how long the app server had been up (how warm it was), whether anyone else was using it, application parameters such as which db servers and in what configuration, how many app servers, and any other interesting things I saw during the run (cpu/mem usage spikes, etc.)
   
2. I would take the output in plain form (see out_graph_results.csv) and convert it to some usable data:
   1. adjust the Timestamp column and create a Time column using the formula =((B2-18000000) / 86400000) + 25569 where B2 is the Timestamp cell
      1. don't really remember where I got this, conversion to epoch time to normal time or something like that
3. Then I would get the usable data I wanted out of it:
   1. put an autofilter on each column
   2. for "label" click the down arrow on the autofilter and choose Text Filter -> Contains for each of the following types of request labels:
      1. /report.do, dwr/call, .css, flow/, download
      2. then when the results were filtered, I would copy everything in the Latency column, thereby getting all the response times for each request type
      3. I think I also filtered the "responseMessage" column to see if there were any errors skewing the response time results
      4. Now that I had the response times for each type, I would take an average of each response time type
4. Finally, take the data from several runs and combine them
   1. Do the above, get averages of several runs, varying 1 condition
      1. type of request (reporting, dwr, downloading)
      2. number of users
   2. graph the run averages to see what type of curve we get
      1. for example, varying the number of users, do runs for 10, 20, 40, 80, 160, 320
      2. space the points on a logarithmic scale and hope that the curve is linear

JMeter quick start guide (part 3)

--------------------------
3. Setting up data and running a test
--------------------------

Update the test file (.jmx) to use the variables from your .csv files.

This is by no means the only way to do this, I've just found it best because the JMeter GUI kind of sucks. There is no Find that I know of and for a big test you may have hundreds of HTTP requests to look through to find the one(s) that contain your variables. It's probably good to look for the one with the login (since it should be near the top of the list of HTTP requests) and see what it looks like in the GUI before and after you open the .jmx and do your search/replace. It should be clear that all the GUI is doing is showing the fields saved in the .jmx and is really pretty transparent as far as 1 request in the .jmx mapping to one HTTP Request screen in the GUI, and all the inputs map directly to xml elements (like Server Name in the GUI mapping to "HTTPSampler.domain").

1. open the test .jmx file with a good editor such as Textpad
2. update the hostname so that you can point the test at a different server at a later date
   1. search the .jmx for your server name or for HTTPSampler.domain. should find a line that looks like: myApp.com
   2. do a global replace of myApp.com with ${HOSTNAME}
      1. hopefully myApp.com will not appear any where else in the .jmx. It shouldn't, but it may be safer to replace something like HTTPSampler.domain">myApp.com with HTTPSampler.domain">${HOSTNAME}
3. update any inputs you gave during the recording session (username, password, report name, etc.)
   1. search the .jmx for the username you put in, let's say it's "myUser"
   2. replace this single instance of myUser with ${IU_01_USER} (this is the name of the USER variable we chose when setting up the .csv file (see earlier in this documentation))
   3. do the same for password
   4. remember when you're searching that any spaces you used in inputs (like the name of a partner you created) will look like:
      1. My+Company+Name
      2. I just avoid spaces all together and use underscores as spaces
   5. when we set up our Thread Group earlier all we set up was username and password but for other simple inputs such as report name it may be a little trickier. For example when I recorded running a simple report (no inputs to the report) I had to search for the report name a couple times to find it (it was selected from a dropdown, not typed in a text field):
      1. I found this block, which is typical:

      2. <elementProp name="reportType" elementType="HTTPArgument"> <boolProp name="HTTPArgument.always_encode">false</boolProp> <stringProp name="Argument.name">reportType</stringProp> <stringProp name="Argument.value">institution-order-report</stringProp> <stringProp name="Argument.metadata">=</stringProp> </elementProp>

         
         
      3. so I would replace this single instance of institution-order-report with the ${IU_01_REPORT_NAME}
   6. for other more complex inputs, it's the same basic idea, for example when I recorded running a report that took several inputs I had to replace several things. Let's say I wanted to keep the report name the same, but parametrize the inputs (institution id, start date...):
      1. After I found the report name, I find the adjacent elements being sent in the HTTP request such as

      2. <elementProp name="institutionId" elementType="HTTPArgument"> <boolProp name="HTTPArgument.always_encode">false</boolProp> <stringProp name="Argument.name">institutionId</stringProp> <stringProp name="Argument.value">8a5c510e2905229e0129055a035f000c</stringProp> <stringProp name="Argument.metadata">=</stringProp> </elementProp> <elementProp name="startTime" elementType="HTTPArgument"> <boolProp name="HTTPArgument.always_encode">false</boolProp> <stringProp name="Argument.name">startTime</stringProp> <stringProp name="Argument.value">12312010</stringProp> <stringProp name="Argument.metadata">=</stringProp> </elementProp>

         
         
      3. so I would replace this single instance of 8a5c510e2905229e0129055a035f000c with ${IU_01_INSTITUTION_ID} and this single instance of 12312010 with ${IU_01_START_TIME}
   7. there are also JMeter-defined variables such as ${_time(YMDHMS)} and ${_threadNum} which you can use inline with your text such as
      1. jmeter_test_${__time(YMDHMS)}
4. verify in the GUI a few things that you updated in the .jmx
   1. in your thread group, look for a request for /j_spring_security_check and click on it (Edit: this is a page name specific to our app, so yours will likely be different)
      
   2. look in the Send Parameter With the Request section and you should see your updated values being send for username and password

Create some .csv data input files

Filenames are those we chose when setting up the .csv files (see earlier in this documentation):

1. update .csv files
   1. Create HOSTNAME.csv
      1. a single line containing your server's address (such as 1.2.3.4 or www.myApp.com)
   2. Create IU_01.csv
      1. Earlier in this documentation when we set up the Thread Group we set up the CSV Data Set Config element to have username and password in this data file. In that case the data would look like (for example):

         1. userJoe, abc123 userSusi, xyz987 userAlan, ijk345

   3. More complex .csv files (for example report name):
      1. You could have the users and the report names in 1 file, making the file look like:

         1. userJoe, abc123, dailyReport userSusi, xyz987, monthlyReport userAlan, ijk345, yearlyReport

         2. If you modify the file to include this new report name column, you would need to
            1. search/replace the actual report name with ${IU_01_REPORT_NAME} in the .jmx as describe above
            2. update the CSV Data Set Config element as described when we set up the Thread Group earlier in this documentation, adding IU_01_REPORT_NAME to the end of the Variable Names list
      2. Or you could have the users and the report names in separate files, using the username, password file from above and a report file (doesn't need to have the same number of lines) which would just look like:

         1. dailyReport weeklyReport monthlyReport quarterlyReport yearlyReport

         2. if you add a new file with the report name, you would need to
            1. search/replace the actual report name with ${IU_01_REPORT_NAME} in the .jmx as describe above
            2. add a new CSV Data Set Config element as described when we set up the Thread Group earlier in this documentation

Setting up the App server:

1. Disable XSS attack prevention:
   1. Our app uses DWR (a Java lib that implements Ajax), and the DWR requests made from JMeter fail due to a cross-site scripting attack prevention feature in Spring(?). To turn the feature off, add the crossDomainSessionSecurity param to the dwr-invoker servlet in web.xml.
   2. web.xml should be somewhere like /opt/ntc/jetty/webapps/root/WEB-INF on the app server. Add the 4 lines that define the corssDomainSessionSecurity init parameter:

   3. <servlet> <servlet-name>dwr-invoker</servlet-name> <servlet-class>org.directwebremoting.servlet.DwrServlet</servlet-class> <init-param id="dwrInvoker-debug"> <param-name>debug</param-name> <param-value>true</param-value> </init-param> <init-param id="dwrInvoker-scriptCompressed"> <param-name>scriptCompressed</param-name> <param-value>false</param-value> </init-param> <init-param> <param-name>crossDomainSessionSecurity</param-name> <param-value>false</param-value> </init-param> </servlet>

      
      

Running a test locally:

1. There are two ways to run:
   1. Through the GUI: go to Run -> Start
      1. produces the JMeter log file: jakarta-jmeter-2.3.4/bin/jmeter.log
      2. produces the output .csv: jakarta-jmeter-2.3.4/data/output/out_graph_results.csv (which is what we specified when setting up the Graph Results element)
   2. Command line (probably a better idea since it requires less resources than the GUI)
      1. cd .../jakarta-jmeter-2.3.4/data (where the .jmx files are)
      2. jmeter -n -t myTestFile.jmx -j output/jmeterlogfile.log -l output/logfile.log
      3. produces the JMeter log file: jakarta-jmeter-2.3.4/data/output/jmeterlogfile.log
      4. produces the output .csv: jakarta-jmeter-2.3.4/data/output/out_graph_results.csv
      5. produces an XML version of the output .csv: jakarta-jmeter-2.3.4/data/output/logfile.csv
2. After the test, move or rename the output file created by the Graph Results element. If you run the test twice, it will simply append to the old output file (and we don't want that). The graph in the Graph Results element will also just append. It's not an extremely helpful graph in my experience.
3. View:
   1. the output .csv (in Excel) and if you're seeing "Non HTTP response code" all over the place you probably have a config issue (or your app is down). For example if you see "Non HTTP response message: ${HOSTNAME}" then your HOSTNAME.csv isn't being read properly
   2. the JMeter log file has lots of good info. You should be able to search for the names of your input .csv files and see whether they were stored or not found.
4. Tweaking:
   1. It may take several tries to get it to run in a way that simulates a user well. If you're running a report, you'll need to check the app logs to see that the user successfully logged in and that report code is running (hopefully in debug mode, telling you what is being executed) and/or check a temp folder where the generated report may be stored
   2. Be sure to tweak the Uniform Random Timer in your Thread Group to slow down the rate that JMeter is firing off requests so that it's consistent with how a human user would use the system.
   3. Check the other Thread Group options (Number of Threads, Loop Count, etc.) to make sure they still make sense with the data set you've defined

Running a test on the JMeter server:

1. The JMeter server is an Amazon image with the name XXXXXXX.net
2. Base dir is ~/jakarta-jmeter-2.3.4
3. Scripts are in ~/scripts
4. Other dirs are set up the same way we have set up our local environment
5. Ftp your .jmx and input .csv or other files to their respective dirs
6. To run:
   1. Putty in as user ntc
   2. cd ~/jakarta-jmeter-2.3.4/data
   3. jmeter -n -t myTestFile.jmx -j output/jmeterlogfile.log -l output/logfile.log

JMeter quick start guide (part 2)

--------------------------
2. Setting up and recording a test
--------------------------

Overview:

A Test Plan is a group of Thread Groups which all run concurrently.

A Thread Group is a set of config elements and series of HTTP Requests (known as Samplers) for a user or group of users to execute in order. If there are several users in the group, each of them can be thought of as a Thread, and in fact each will run in its own java thread. I typically break Thread Groups down into groups of users of a specific type exercising a specific part of the application. For example: Inst users that each do 1 request or Partner users that visit several screens and exercise DWR functions (sort/filter lists) or Admin users that each run 3 reports.

In our setup, a Thread Group typically has a CSV Data Set Config, a Cookie Manager, a Uniform Random Timer, and the HTTP Requests.

Setting up a Test Plan:

1. right click on your Test Plan, Add -> Config Element -> CSV Data Set Config
   1. Name: Hostname Config
   2. Filename: ../data/input/HOSTNAME.csv
   3. Variable Names: HOSTNAME
   4. Delimiter: a plain comma char
   5. Recycle on EOF?: I usually set this to true and use the Thread Group's Number of Threads and Loop Count to control how the test runs
   6. Stop thread on EOF?: again, I usually use the Thread Group to control this
2. right click on your Test Plan, Add -> Listener -> Graph Results
   1. Filename: ../data/output/out_graph_results.csv
   2. on the Graph Results screen, click Configure
      1. uncheck any box with (XML)
      2. check: save field names, save active thread counts, save assertion failure message

Setting up a Thread Group in JMeter (we'll use my Inst User (IU) group 01 as an example):

1. right click on your Test Plan, Add -> Thread Group
   1. give it a descriptive name and description in comments field
   2. Number of Threads: If you have 10 users in your input .csv, this # can be up to 10.
   3. Rampup: If it's a heavy duty test (20+ min), not bad to make it ramp for a couple minutes.
   4. Loop Count: Only tricky thing here is if you have 10 users in your .csv and you enter 3 in the Number of Threads and 2 in Loop count, users 1-6 in your .csv will each run the HTTP Requests 1 time. Another thing is, in this scenario for example, it will try to keep 3 threads alive, meaning if threads 1 and 2 finish and thread 3 is slow, threads 4 and 5 will kick off and run while 3 is also running.
2. right click on your Thread Group, Add -> Config Element -> HTTP Cookie Manager
   1. done with cookie manager
3. right click on your Thread Group, Add -> Config Element -> CSV Data Set Config
   1. Filename: something like "../data/input/IU_01.csv"
   2. Variable Names: a comma separated list of your var names such as IU_01_USER,IU_01_PASS
   3. Delimiter: a plain comma character
4. right click on your Thread Group, Add -> Timer -> Uniform Random Timer
   1. I'm not positive this is the best way to do this, but I believe it is: Let's say you're running a report. When you record the test and you're actually going through the process of logging in, running the report, logging out, let's say it takes you 3 minutes. Now when you run the test, it will run the requests back-to-back, making it take 20 seconds or so. Play with the Timer's Random Delay Maximum field and try to find a number that makes the test take about 3 minutes so that it simulates a real user. Mine are typically between 1500-2500ms
5. save your Test Plan in the data folder (jakarta-jmeter-2.5.1/data) and give it a descriptive name, not for the Thread Group but for the whole Test Plan!

Set up HTTP Proxy (for recording) in JMeter:

1. right click Workbench -> Add -> Non-test Elements -> HTTP Proxy Server
2. if you have a specific Thread Group that you want to save the recording to, choose it in the Target Controller dropdown
3. check Attempt HTTPS Spoofing
4. under URL Patterns to Exclude, click Add
5. double-click the text area just created, enter .*google.* and press enter (don't want requests to google fouling up our response time metrics!)
   
6. save the test plan

Record from Firefox (IE is similar):

1. start Firefox
2. go to Tools -> Options
3. click Advanced at the top
4. click the Network tab
5. under Connections click Settings
6. click the Manual Proxy Config radio button
7. in HTTP Proxy enter localhost, port 8080
8. check "Use this proxy server for all protocols"
9. clear out the "No proxy for" text area
10. click ok a few times
11. if you want to simulate a user who has not been to the site before (has no JS cached), go to Tools -> Clear Recent History and clear it out
12. in the browser address bar now instead of using https... you'll use http... when entering the app URL
13. in JMeter, click Start at the bottom of the HTTP Proxy Server config screen
14. log in, run a report, do whatever it is you want to record, log out
15. in JMeter, click Stop at the bottom of the HTTP Proxy Server config screen
16. undo the Proxy settings in Firefox if you're done
17. save the Test Plan

JMeter quick start guide (part 1)

Everyone seems to love JMeter, but I found the learning curve quite steep. It seems to be a blank palette with lots of "this is how this works" kind of verbiage but not a whole lot of "this is how you use this tool." So with that, here's a howto I wrote for my company. Any confidential info will be replaced with XXXXXXX.

Let me STRESS that this is how I did it but may or may not be how most people do it or at all the best way to do it. I am completely open to suggestion and constructive criticism because I felt like I did this from scratch. I will post these in 4 parts as I did on my company's wiki.

--------------------
0. Overview
--------------------

Overview:

JMeter is a load test suite (unfortunately) written in java. We use it to gather performance data based on realistic loads of various types (reporting, ordering, data format translation, DWR, admin functions, etc.) by various user types. The basic gist is that you

1. record a series of http requests into a .jmx file
2. replace the hardcoded values (server names, user names, etc.) with variables such as ${INST_USER_01}
3. create input .csv files with values for those variables
4. feed the .csv and .jmx files to jmeter to run the load

The tester should watch the app's log and check that (for example) the reports have been run to make sure the requests are running correctly. If the app changes significantly, the tests may need to be re-recorded. Hopefully this will be rare.

JMeter is highly configurable and has a steep learning curve. The user manual (http://jakarta.apache.org/jmeter/usermanual/index.html) is quite good but I'll try to write the steps in a series of child pages.

Notes:

Be careful of /tmp filling up on the app server.

--------------------
1. Setting up JMeter
--------------------

more likely than not, you'll be using our JMeter box to run tests, but you'll need it on your local machine to develop:

1. download JMeter
   1. our copy (version 2.3.4):
   2. online: http://jakarta.apache.org/site/downloads/downloads_jmeter.cgi
2. to install, just unzip into a folder on your machine
3. create our folders
   1. jakarta-jmeter-2.3.4/data
      1. where the .jmx test files will go
   2. jakarta-jmeter-2.3.4/data/input
      1. where the .csv (user and hostname variables), .zip (bulk upload) files will go
   3. jakarta-jmeter-2.3.4/data/output
      1. output log and .csv
   4. jakarta-jmeter-2.3.4/data/old
      1. a place to put old output
4. config files
   1. bin/jmeter.bat
      1. I set the Heap as high as it would go on my local machine (think this is my only customization):
      2. set HEAP=-Xms512m -Xmx1400m