Automation Strategy (Email): Putting Intelligence into Your Library Methods

Summary

This is an email I sent to my team who had written some if statements which were dependent on the customer in use (customers could customize their UIs to some extent).  I let them know that since several customers will have similar UIs, we don't need to do an if based on each customer ID, we can put some intelligence into the method so it can decide what to do based on what the UI looks like.

In general, I'm not interested in testing that the UI is correct and that the test data is set up correctly - I'm interested in testing the logic, e.g. add to cart, change quantity, make a purchase.  That statement is relevant here because let's say customer 1234's should be set up to use UI version A and there's a bug in UI version A.  The test should fail.  But if the test data has customer 1234 set up to use UI version B, the automation will happily use UI version B and may pass.  But I would argue that's a data setup problem, not an automation problem, and would be a problem even if the test were run manually.

Email

Currently in ProductFlow.java we have code that looks like this:

 public void addProductToShoppingCart(CheckOutProduct checkOutProduct, String catalog) {
     if (catalog.equals(StaticData.US_CATALOG_1234) {
         //do one thing
     } else {
         //do a different thing

     }

If we use lots of manufacturers, this list will get big.  This method addProductToShoppingCart() does not have much intelligence in it, and theoretically the manufacturer's page could change.  Something we can do instead of blindly relying on the manufacturer ID is to make the method more intelligent.  The method could analyze the page, such as:

 public void addProductToShoppingCart(CheckOutProduct checkOutProduct, String catalog) {
     if (navigationLinkA.isPresent()) {
         //do one thing
     }
     if (navigationLinkB.isPresent()) {
         //do a different thing
     }

An example of this would be in the selling portal – some customers have a side navigation link with text "Part Numbers" and other have a link that has text "List by Part Number"

In fact, now that I think about it, a better solution would be to have a method called "navigateToPartList()" and put these if statements in there, then all addProductToShoppingCart() would do is call navigateToPartList()... 

As I've said many times before, let's keep pushing that work farther down the road (make someone else do it), and have lots and lots of very simple, reusable methods.

As always, this is not the best solution in all cases.  Using the customer ID is perfectly fine sometimes, this is just an alternative way of doing things.

Automation Strategy (Email): We Will Achieve Efficiency Through Correctness, Not Speed

I just thought this was important enough to make a post about.  At first I thought it would be self-explanatory, but in case it isn't - I'm talking about team efficiency, not processor efficiency, although the case could be made there as well.

This is a team of contractors who, like most contractors, wanted to pump out tests and executions.  I consistently ask them to slow down and write good code so that our pull requests go faster and our code requires less debugging during automation execution.

---

From: JW

Date: Monday, November 26, 2018 at 1:20 PM

To: AP, VG

Subject: Re: Test_Automation(22-Nov-2018)

Team,
Please always pay attention to details.  Our Pull Request numbers and SQA Project numbers are very close.  Below it says PR 475 but should be SQA-475.  Not a big deal, but just go a little slower - we will become the most efficient by prioritizing correctness over speed.  The same is true for writing clear, understandable code.  We need to think more about the next developer than we do about ourselves.  If it takes you extra time to save your teammate time, please do it - a week later you may be the one receiving this gift!


JW
Automation Lead

---

From: AP
Date: Thursday, November 22, 2018 at 10:45 AM
To: JW
Subject: Test_Automation(22-Nov-2018)

Hello Team,
As per plan of action, team worked on automated suite execution and failure analysis of 18.12 release.

AP:         

• Worked on failure analysis of OR and fulfiller-15569 automated regression tests(SQA-469)
• Updated the Wiki page for 11 known issues of 18.12 release. 

VG:

• Reviewed PR 475 and updated comments in JIRA for it
• Reviewed PR-419 and updated comments in JIRA for it
• Created PR-423,PR-424 for SQA-453 and completed it

Tomorrow's plan of action:- To continue work on fixing the failed test-script and handling prescripts and further work on SQA board tickets.

Thanks!

Automation Strategy (Email): Method Naming - Do What You Promise

Team,

I have always made clear how critically important method names are.  Sometimes the difference is very big and sometimes is it more subtle.  I just want to point out a couple of examples I've just seen.

1.

A pretty big one is

verifyUserExistence()

which looks to see if a user exists, but if it does not, it creates a new user.  Creating a new user has NOTHING to do with verifying a user's existence.  I have created an issue to refactor the name to verifyOrCreateUser()

2.

adminViewFulfillerUsersPage.enterNewUserDetails()

I would expect this method to enter text fields on a page.  What it actually does is enter all the text fields, then click the Create User button.  This is a subtle difference, but in some refactoring, Vibhav renamed it to

adminViewFulfillerUsersPage.createAdminFulfillerUser()

which I believe is a better name, because this method does more than simply entering details.

Every time you write a new method, give serious thought to what its name should be.  The next developer to come along will waste a lot of time and be upset with you if your method says one thing but actually does another (or does more than what it says).  It would be like if you asked someone "Please get me a drink" and they said "No problem," then they got you a drink, drank all of it, and handed you an empty cup.  Don't be that guy!

Note that we didn't change the name from

enterNewUserDetails()

to 

enterNewUserDetailsAndClickCreateButton()

instead, we go up a level in abstraction and just called it

createAdminFulfillerUser()

This helps avoid creating methods with very long names.  If you have any questions or would like suggestions for method names, please ask.

Thanks-

Automation Strategy: Externalize All Your Strings

Motivation

We need to externalize all the strings we use in our locators, such as "User order number" or validation methods, such as "Order Created."  Why?

1. It'll make your code more maintainable if that string changes
2. It keeps things organized
3. Internationalization (i18n)
4. Multi-platform automation

From the outset, you might as well plan for i18n by including a switch for locale (not language). Language is "English."   Locale is "American English" and looks like "en_US" or "en-US" depending on who you ask.  Google it.  I like to model the strings exactly as they are, including capitalization and punctuation:

This works well for automating small to medium apps that themselves maximize code reuse.  But for big (or legacy) applications it may not be the best approach due to non-standardization, your mileage may very.

And for multi-platform automation... (there is a lot more work involved, but this is part of the deal...)

Ok I'm going to be honest right now. I wrote the part above tonight and this next part 2 years ago and I don't even know what language this is. The filenames are .cs but I don't even know C# well enough to know if you can define methods like this, I might have just used .cs because gist colorization was good.  Hmm, now I'm thinking it's just pseudocode.

Anyway, the last thing I'll say about externalizing strings is about dynamic strings.
Here's option 1, strings live in the data file and you compose them in the page object class:

This is fine, but will require everyone who uses those strings to do all that string concatenation themselves, which is code duplication. It will also become a problem when the translation is weird. Maybe in American English we'd say "Order number 404 is on its way!" but in Canadian English they'd say "404 is the number of the order comin' at ya, eh!" If this were the case, our stringA + orderID + stringB wouldn't work, or would be wonky. Let's just write methods to generate those strings instead, and notice how it's used in the page class:

And you could even get really crazy and do kind of a mix, which lets you have both the individual strings as well as the pre-composed dynamic strings. It just depends on what you need for your situation:

Automation Strategy: Keep Going With Page Objects (Don't Stop at Locators)

Stopping at the Simplest Implementation of POM

I don't love the Page-Object Model (POM).  Actually, I just don't love the way many teams implement it.  When they stop at giving their page elements locators, they end up with elements that look like

Reusing Page Objects in Other Page Objects

It's these last few that get me. Any time I see redundant code it makes me cringe. Let's level-up:


This can probably only be achieved if you've wrapped your Selenium tool's WebElement and WebDriver class. Hopefully you did this. It will take a little cleverness and adding some constructors, but it should be doable.

Dynamic Elements Using Methods, Not Variables

I find myself kicking the can down the road, or pushing the work down to a lower level like this all the time. "Make someone else do the work" I say. If I'm in the middle of writing a method and I think for one second that what I'm about to write could be used somewhere else, I try to encapsulate that work in a method. Better yet, I imagine what I'd pass into that method, what it would return to me, and continue writing my current method as if that helper method exists and keep going. Later on I'll write the helper methods into existence. 

In this case, while we're writing createAndViewOrder(), we don't want to worry about how we find the orderConfirmation element, we just want to assume it works and use it, so let's move it out of the method.  Before:

After:


Does this buy you a lot right now? Not really. But

1. it might buy you something down the road during a refactor
2. it makes your functional method easier to read
3. it literally takes 15 extra seconds

Summary

It goes to a frame of mind that is good to get into when writing automation code - avoid UI specifics in functional methods, because you're likely to re-use that UI element again. There's a LOT of patterns when writing automation code from how your tests are constructed to how you find page objects to how you interact with the app, so you really have to be on guard and work against code duplication, this is just one of the ways.

Automation Strategy: Positively Do Negative Testing With should_fail()

Motivation

An old adage is "don't code for the exception," which is along the lines of Keep It Simple Stupid. This doesn't mean you don't expect and handle exceptions, bad data, etc., but that you should try to avoid extensive, complex coding for very rare cases.

I try to employ this in automation development just like any other software development, therefore when I write higher-level methods, I write them for the 95% case (the rule), and avoid the 5% case (the exception). For example, if I'm writing a login() method and after logging in, there's a popup that I will dismiss in 95% of my test cases, I will include the dismissal in the login() method. If I'm using a language that supports optional parameters, God forbid I have to work in a language that doesn't, maybe I'll take an optional parameter dismiss=True.

I'm actually quite proud of this should_fail() stuff.  I always aim to simplify the test scripts so they can be written by less technical people (hey, isn't that the point of cucumber/gherkin blah blah blah) and should_fail() aids in that as well as keeping your tests user-centric instead of UI-centric. This way, you can use your great Flow methods even though they feel like they're designed for positive-path testing.

For this example, we're logging in, then verifying we're on the Home page.  In this system, if the "passed" variable isn't true by the time the test is over, the test is considered to have failed.
Here's the positive test:

Now For Negative Testing

Everything's hunky dory, which now that I type it is a very strange phrase. For the simple negative test, we'll try to login and then verify that we're still on the login page... that we have not advanced to the homepage (we could also look for an error message such as "User not found"). We can't use login() because it will try to dismiss that modal after we login and throw an ElementNotFoundException or something like that, so we'll have to use clicks:

Using Exceptions the Simple Way

Well that doesn't look too bad, but it was very-UI centric, and as our automation gets more complex, maintainability of these negative tests is going to become a factor. If all our positive testing is done using Flows, we've thrown it out the window and reverted to using page objects. Ok, let's actually try it with login() and just catch the exception:

Using should_fail()

That works, but it's a lot of code. Let's hide and generalize that code in our should_fail() method:


Ok, there's definitely some syntax to learn there, but not too much. Let's see how should_fail() works:

One thing you might notice is the hardcoding of the error messages. Those would be in some kind of dataStrings.rb file or something. The other is that I'm throwing around exceptions here quite a bit. This is not necessarily something you can shove into a pre-existing system that doesn't use exceptions very much, or doesn't use custom exception messages. All my wrapper click() and find() and type_text() and everything else methods throw exceptions with clear, relevant messages in order to make debugging easier.

Summary and Tying in Flows

So again, the whole point of this is so you can write your negative tests using Flow methods. Let's say in the example below that returning an order is a multi-step process. If we were to write this negative test using the page objects only similar to my Test_Negative_Using_Clicks() above, we would be reverting to a long test that was very tied to the UI. So by using positive testing of negative tests, we can stay user-centric and high-level, even if your task is complex. In the example below we're going to try to return an non-returnable item:


The last thing I'll say is the error I'm looking for here is very high-level. Sometimes it's difficult or tedious to make your Flow methods do this, so maybe in this example, we would have tried to click a button as part of the return process but it wasn't there or was disabled, etc. In that case, we'd just have to catch the more generic "... does not exist" exception, and this does tie us to the UI. The usefulness of this system, like all automation, depends on the context of the system you're testing.

Automation Strategy: Flow vs Page Objects Part 2 - Test Data Abstraction

In my first post on Flow vs Page Objects, it became clear that as we move up in layers of abstraction, our tests get shorter, but the data required to execute those tests remains the same. If we were just using page objects, we could do something simple such as...

Using Flows

...but again, we want to focus on the user, and if you asked the user what they do on this page, they wouldn't say "This is the page where I put in my first name, then last name, then street, then city, then state, then zip code," they would say "This is where I put in my address." So let's do that:

Maintenance

This is going to get overwhelming quickly, on top of which it's not maintainable. You might think the structure of a user's address doesn't change much, but maybe it's a voting website and they now require you to put in your state representative district number. Yeah, it can and will change. It needs to be maintainable. Now all our address tests need to be updated.

If you ask a user what their address is, they don't say "My street is 123 Main St. My city is Austin. My state is Texas," they say "My address is 123 Main St, Austin, Texas." So instead of passing in all the pieces of information that form an address, let's just pass in an address:

Maintenance Moves from Tests to the Data Source and Related Classes

The trick here of course is that our data source is going to have to get smarter. That's up to you, whether it's coming from a file, a DB, XML, JSON, who knows. The point is, when we add the state rep district number, the things that have to change are:

1. The data itself.  And yes, if we have 1 data file for every test, then yes, we'll have to update a bunch of files.  This will have to be done in any system, and if you reuse the same data file in N tests, then it won't be a ton of files to update.
2. The code to get the data from the file.  If well designed, this should be trivial, and if very well designed, this might not have to change at all.
3. The classes or containers (such as Structs in C++) that hold the data. This should be trivial.
4. The PO class needs a new object for the new UI element. Trivial.
5. The Flow code that actually enters the data into the UI using that PO. Trivial.

Let's say our data file looks like this, then we add the district number to the end:

and our code for holding the data looks like this, then all we have to do is update that last line:

and our Page class (the Flow doesn't do much in this trivial case):

Payoff

So that's it. You're going to have to update the data and how the data gets input into the UI regardless, there's no magic bullet for those. But this way, your tests don't change. Again, this is a simple example. The more complex it gets or the bigger a change to the application's workflow, the more this extra abstraction will pay off.

The last thing I'll say is this: notice we're updating the address, but the flow isn't updateAddressFlow, it's just addressFlow. I encourage grouping related functionality together. In fact, where we're doing this now, for the smaller of our dozen or so applications, there's just a single flow for each application. How you break that up is a trade-off you make between having tons of code files and having tons of code in just a few files.

Automation Strategy: Flow vs Page Objects Part 1 - or - If You Think Page Objects are Enough, Think Harder

Motivation

The Page Object Model (POM) has been the de facto standard in UI automation using Selenium for years now.  Here* is a typical primer on the subject.  This primer does a perfectly reasonable job of showing how people normally use POM, and it also clearly shows the reason that using POM the way I've seen a lot of people use it is insufficient for writing maintainable tests.

The problem is illuminated in the author's 5th reason for using POM: "Any change in UI can easily be implemented, updated and maintained into the Page Objects and Classes."  If you've done this for a while, you'll notice what is not said here - that a change in business process or higher-level user interaction with the site cannot be easily implemented in our automation.  For that type of change, if we were to only use POM, we'd have to update all of the related tests. That is to say, this style of POM is sufficient for encapsulation and maintainability of UI elements and structure, but we need something higher-level to encapsulate and maintain business processes.

Typical POM

As a trivial example, let's say you have this PO class and test:

You're going to have hundreds of other tests that contain this same login code of course.  What happens when the id of the username field changes?  With POM, you're golden.  You just update the username element on your LoginPage class.  But what happens when there's a new step in logging in?  Now all of your tests have to change to look like:

As an automation developer, you can be certain that your site will change - and not just in rearrangement or renaming elements on a page. Some people will say "Hey, I've got a method called login() in my LoginPage class, so that's all I'll have to update." Well kudos, you're a big step closer, let's check it out. We have the login() method to our LoginPage class, so all we do is add the element and the line in the login() method. While we're at it, we'll put the goto() in there. That way our test will be super clean and maintainable:

You will be good to go for this trivial example. But what about when your company decides to change a business process in your app? Let's say you want to add a step in a checkout process such as buying a couch. In the past, you had the code below... you were smart so you even wrapped clicking submitButton with a method, not wanting to mix the use of PO methods like loginPage.login() and use of the PO elements themselves like loginPage.submitButton.click():

The Issue...

The problem is that we're still modeling the interaction with the UI - we're focused on the website. What we should be modeling is what the user wants to do. The user doesn't want to select from a dropdown on pageA and click some buttons on pageB, they're trying to buy a couch, and we're not encapsulating that anywhere, except arguably in the test. That's too late, because we're going to have dozens of really similar tests where the user just wants to do a tiny thing different, such as choosing express shipping vs. regular shipping, so all that test code is going to be duplicated, and we're not going to be DRY at all.

In this example our app changed, and now we have an "assembly and delivery" step. All our tests have to change again:

Flow classes

The Flow class is intended to do this encapsulation and abstraction for us.  It models what the user is trying to accomplish (user stories?) when they're using your application.  Let's check it out:


And now your tests can model the use cases, and they simply looks like:

So maybe the title of this post shouldn't be Flow vs Page Objects, because both layers are useful and work together. The POs encapsulate the UI, and the Flow encapsulates the business rules. Then the tests simply encapsulate the use cases. Honestly, if your app is small enough, like a mobile app, you can do both in the Flow class. I've done this successfully, but a UI refactor probably would have cost me more than if I'd done a proper PO layer.

Looking at the test case, you probably see the potential problems:

1. The more work we make our methods do, the more data we have to pass in to let them know how to do it. For this, we'll have to start consolidating data elements into their logical groups. I'll talk about that in the next post here.
2. Using flows works great for positive tests, but what about negative tests?  I'll talk about that in this post.

Continue...

The next post in this series is Automation Strategy: Flow vs Page Objects Part 2

*In case the site I referenced is deleted, here is a webArchive of it

Automation Trick: "Does image exist" across screen sizes when you have no strong identifier

I have pulled so many automation tricks out of my *&#$ I should have started writing them down years ago.  Maybe someone should make a repository of them.  Anyway, I was trying to verify that our client's logo exists on particular pages of their mobile app.  I reluctantly started going down the dumb path of

Get all the android.widget.ImageView elements and see if size is in the array of possible sizes:
[{'width': 145, 'height': 43},      # tab 4
 {'width': 217, 'height': 64},      # galaxy s3
 {'width': 217, 'height': 64},      # nexus 4
 {'width': 217, 'height': 64},      # nexus 7
 {'width': 298, 'height': 88},      # galaxy s5
 {'width': 298, 'height': 88},      # note 3
 {'width': 298, 'height': 88},      # nexus 5
 {'width': 298, 'height': 88},      # htc One
 {'width': 348, 'height': 103},     # note 5
 {'width': 348, 'height': 103}      # nexus 6
]

and then I realized that the height/width ratio should be the same.  If it's not, we have an even bigger problem of their logo being stretched in a way that their brand manager is going to have a real problem with...

All these ratios are between 0.2949 .. 0.2965, so I'll just round a little (for future devices) to 0.2945 .. 0.297 which is a .25% margin and hope that no other image falls in this ratio.  Should do the trick.

JMeter quick start guide (part 4)

--------------------------
4. Understanding results
--------------------------

1. As a history of each run, I recorded information about each run in a file called "description of run.txt" in which I'd describe things like how long the app server had been up (how warm it was), whether anyone else was using it, application parameters such as which db servers and in what configuration, how many app servers, and any other interesting things I saw during the run (cpu/mem usage spikes, etc.)
   
2. I would take the output in plain form (see out_graph_results.csv) and convert it to some usable data:
   1. adjust the Timestamp column and create a Time column using the formula =((B2-18000000) / 86400000) + 25569 where B2 is the Timestamp cell
      1. don't really remember where I got this, conversion to epoch time to normal time or something like that
3. Then I would get the usable data I wanted out of it:
   1. put an autofilter on each column
   2. for "label" click the down arrow on the autofilter and choose Text Filter -> Contains for each of the following types of request labels:
      1. /report.do, dwr/call, .css, flow/, download
      2. then when the results were filtered, I would copy everything in the Latency column, thereby getting all the response times for each request type
      3. I think I also filtered the "responseMessage" column to see if there were any errors skewing the response time results
      4. Now that I had the response times for each type, I would take an average of each response time type
4. Finally, take the data from several runs and combine them
   1. Do the above, get averages of several runs, varying 1 condition
      1. type of request (reporting, dwr, downloading)
      2. number of users
   2. graph the run averages to see what type of curve we get
      1. for example, varying the number of users, do runs for 10, 20, 40, 80, 160, 320
      2. space the points on a logarithmic scale and hope that the curve is linear

JMeter quick start guide (part 3)

--------------------------
3. Setting up data and running a test
--------------------------

Update the test file (.jmx) to use the variables from your .csv files.

This is by no means the only way to do this, I've just found it best because the JMeter GUI kind of sucks. There is no Find that I know of and for a big test you may have hundreds of HTTP requests to look through to find the one(s) that contain your variables. It's probably good to look for the one with the login (since it should be near the top of the list of HTTP requests) and see what it looks like in the GUI before and after you open the .jmx and do your search/replace. It should be clear that all the GUI is doing is showing the fields saved in the .jmx and is really pretty transparent as far as 1 request in the .jmx mapping to one HTTP Request screen in the GUI, and all the inputs map directly to xml elements (like Server Name in the GUI mapping to "HTTPSampler.domain").

1. open the test .jmx file with a good editor such as Textpad
2. update the hostname so that you can point the test at a different server at a later date
   1. search the .jmx for your server name or for HTTPSampler.domain. should find a line that looks like: myApp.com
   2. do a global replace of myApp.com with ${HOSTNAME}
      1. hopefully myApp.com will not appear any where else in the .jmx. It shouldn't, but it may be safer to replace something like HTTPSampler.domain">myApp.com with HTTPSampler.domain">${HOSTNAME}
3. update any inputs you gave during the recording session (username, password, report name, etc.)
   1. search the .jmx for the username you put in, let's say it's "myUser"
   2. replace this single instance of myUser with ${IU_01_USER} (this is the name of the USER variable we chose when setting up the .csv file (see earlier in this documentation))
   3. do the same for password
   4. remember when you're searching that any spaces you used in inputs (like the name of a partner you created) will look like:
      1. My+Company+Name
      2. I just avoid spaces all together and use underscores as spaces
   5. when we set up our Thread Group earlier all we set up was username and password but for other simple inputs such as report name it may be a little trickier. For example when I recorded running a simple report (no inputs to the report) I had to search for the report name a couple times to find it (it was selected from a dropdown, not typed in a text field):
      1. I found this block, which is typical:

      2. <elementProp name="reportType" elementType="HTTPArgument"> <boolProp name="HTTPArgument.always_encode">false</boolProp> <stringProp name="Argument.name">reportType</stringProp> <stringProp name="Argument.value">institution-order-report</stringProp> <stringProp name="Argument.metadata">=</stringProp> </elementProp>

         
         
      3. so I would replace this single instance of institution-order-report with the ${IU_01_REPORT_NAME}
   6. for other more complex inputs, it's the same basic idea, for example when I recorded running a report that took several inputs I had to replace several things. Let's say I wanted to keep the report name the same, but parametrize the inputs (institution id, start date...):
      1. After I found the report name, I find the adjacent elements being sent in the HTTP request such as

      2. <elementProp name="institutionId" elementType="HTTPArgument"> <boolProp name="HTTPArgument.always_encode">false</boolProp> <stringProp name="Argument.name">institutionId</stringProp> <stringProp name="Argument.value">8a5c510e2905229e0129055a035f000c</stringProp> <stringProp name="Argument.metadata">=</stringProp> </elementProp> <elementProp name="startTime" elementType="HTTPArgument"> <boolProp name="HTTPArgument.always_encode">false</boolProp> <stringProp name="Argument.name">startTime</stringProp> <stringProp name="Argument.value">12312010</stringProp> <stringProp name="Argument.metadata">=</stringProp> </elementProp>

         
         
      3. so I would replace this single instance of 8a5c510e2905229e0129055a035f000c with ${IU_01_INSTITUTION_ID} and this single instance of 12312010 with ${IU_01_START_TIME}
   7. there are also JMeter-defined variables such as ${_time(YMDHMS)} and ${_threadNum} which you can use inline with your text such as
      1. jmeter_test_${__time(YMDHMS)}
4. verify in the GUI a few things that you updated in the .jmx
   1. in your thread group, look for a request for /j_spring_security_check and click on it (Edit: this is a page name specific to our app, so yours will likely be different)
      
   2. look in the Send Parameter With the Request section and you should see your updated values being send for username and password

Create some .csv data input files

Filenames are those we chose when setting up the .csv files (see earlier in this documentation):

1. update .csv files
   1. Create HOSTNAME.csv
      1. a single line containing your server's address (such as 1.2.3.4 or www.myApp.com)
   2. Create IU_01.csv
      1. Earlier in this documentation when we set up the Thread Group we set up the CSV Data Set Config element to have username and password in this data file. In that case the data would look like (for example):

         1. userJoe, abc123 userSusi, xyz987 userAlan, ijk345

   3. More complex .csv files (for example report name):
      1. You could have the users and the report names in 1 file, making the file look like:

         1. userJoe, abc123, dailyReport userSusi, xyz987, monthlyReport userAlan, ijk345, yearlyReport

         2. If you modify the file to include this new report name column, you would need to
            1. search/replace the actual report name with ${IU_01_REPORT_NAME} in the .jmx as describe above
            2. update the CSV Data Set Config element as described when we set up the Thread Group earlier in this documentation, adding IU_01_REPORT_NAME to the end of the Variable Names list
      2. Or you could have the users and the report names in separate files, using the username, password file from above and a report file (doesn't need to have the same number of lines) which would just look like:

         1. dailyReport weeklyReport monthlyReport quarterlyReport yearlyReport

         2. if you add a new file with the report name, you would need to
            1. search/replace the actual report name with ${IU_01_REPORT_NAME} in the .jmx as describe above
            2. add a new CSV Data Set Config element as described when we set up the Thread Group earlier in this documentation

Setting up the App server:

1. Disable XSS attack prevention:
   1. Our app uses DWR (a Java lib that implements Ajax), and the DWR requests made from JMeter fail due to a cross-site scripting attack prevention feature in Spring(?). To turn the feature off, add the crossDomainSessionSecurity param to the dwr-invoker servlet in web.xml.
   2. web.xml should be somewhere like /opt/ntc/jetty/webapps/root/WEB-INF on the app server. Add the 4 lines that define the corssDomainSessionSecurity init parameter:

   3. <servlet> <servlet-name>dwr-invoker</servlet-name> <servlet-class>org.directwebremoting.servlet.DwrServlet</servlet-class> <init-param id="dwrInvoker-debug"> <param-name>debug</param-name> <param-value>true</param-value> </init-param> <init-param id="dwrInvoker-scriptCompressed"> <param-name>scriptCompressed</param-name> <param-value>false</param-value> </init-param> <init-param> <param-name>crossDomainSessionSecurity</param-name> <param-value>false</param-value> </init-param> </servlet>

      
      

Running a test locally:

1. There are two ways to run:
   1. Through the GUI: go to Run -> Start
      1. produces the JMeter log file: jakarta-jmeter-2.3.4/bin/jmeter.log
      2. produces the output .csv: jakarta-jmeter-2.3.4/data/output/out_graph_results.csv (which is what we specified when setting up the Graph Results element)
   2. Command line (probably a better idea since it requires less resources than the GUI)
      1. cd .../jakarta-jmeter-2.3.4/data (where the .jmx files are)
      2. jmeter -n -t myTestFile.jmx -j output/jmeterlogfile.log -l output/logfile.log
      3. produces the JMeter log file: jakarta-jmeter-2.3.4/data/output/jmeterlogfile.log
      4. produces the output .csv: jakarta-jmeter-2.3.4/data/output/out_graph_results.csv
      5. produces an XML version of the output .csv: jakarta-jmeter-2.3.4/data/output/logfile.csv
2. After the test, move or rename the output file created by the Graph Results element. If you run the test twice, it will simply append to the old output file (and we don't want that). The graph in the Graph Results element will also just append. It's not an extremely helpful graph in my experience.
3. View:
   1. the output .csv (in Excel) and if you're seeing "Non HTTP response code" all over the place you probably have a config issue (or your app is down). For example if you see "Non HTTP response message: ${HOSTNAME}" then your HOSTNAME.csv isn't being read properly
   2. the JMeter log file has lots of good info. You should be able to search for the names of your input .csv files and see whether they were stored or not found.
4. Tweaking:
   1. It may take several tries to get it to run in a way that simulates a user well. If you're running a report, you'll need to check the app logs to see that the user successfully logged in and that report code is running (hopefully in debug mode, telling you what is being executed) and/or check a temp folder where the generated report may be stored
   2. Be sure to tweak the Uniform Random Timer in your Thread Group to slow down the rate that JMeter is firing off requests so that it's consistent with how a human user would use the system.
   3. Check the other Thread Group options (Number of Threads, Loop Count, etc.) to make sure they still make sense with the data set you've defined

Running a test on the JMeter server:

1. The JMeter server is an Amazon image with the name XXXXXXX.net
2. Base dir is ~/jakarta-jmeter-2.3.4
3. Scripts are in ~/scripts
4. Other dirs are set up the same way we have set up our local environment
5. Ftp your .jmx and input .csv or other files to their respective dirs
6. To run:
   1. Putty in as user ntc
   2. cd ~/jakarta-jmeter-2.3.4/data
   3. jmeter -n -t myTestFile.jmx -j output/jmeterlogfile.log -l output/logfile.log