Automation Strategy: Positively Do Negative Testing With should_fail()

Motivation

An old adage is "don't code for the exception," which is along the lines of Keep It Simple Stupid. This doesn't mean you don't expect and handle exceptions, bad data, etc., but that you should try to avoid extensive, complex coding for very rare cases.

I try to employ this in automation development just like any other software development, therefore when I write higher-level methods, I write them for the 95% case (the rule), and avoid the 5% case (the exception). For example, if I'm writing a login() method and after logging in, there's a popup that I will dismiss in 95% of my test cases, I will include the dismissal in the login() method. If I'm using a language that supports optional parameters, God forbid I have to work in a language that doesn't, maybe I'll take an optional parameter dismiss=True.

I'm actually quite proud of this should_fail() stuff.  I always aim to simplify the test scripts so they can be written by less technical people (hey, isn't that the point of cucumber/gherkin blah blah blah) and should_fail() aids in that as well as keeping your tests user-centric instead of UI-centric. This way, you can use your great Flow methods even though they feel like they're designed for positive-path testing.

For this example, we're logging in, then verifying we're on the Home page.  In this system, if the "passed" variable isn't true by the time the test is over, the test is considered to have failed.
Here's the positive test:

	class LoginPage()
	username = driver.TextField(:id, "username")
	password = ...
	submit = ...
	close_welcome_modal = ...
	def login(uname, pwd)
	username.type_text(uname)
	password.type_text(pwd)
	submit.click()
	close_welcome_modal.click()
	def verify_on_page()
	# assert that our username element is present, else throw an exception with Error message
	assertTrue (username.present(), "Error: we are not on the Login page")
	class HomePage()
	def verify_on_page()
	assertTrue (main_menu.present(), "Error: we are not on the Home page")
	class TestClass()
	def Test_Positive()
	lp = LoginPage()
	hp = HomePage()
	lp.login(data.uname, data.pwd) # valid user/pwd
	hp.verify_on_page() # we're on the homepage
	passed = true

view raw should01.rb hosted with ❤ by GitHub

Now For Negative Testing

Everything's hunky dory, which now that I type it is a very strange phrase. For the simple negative test, we'll try to login and then verify that we're still on the login page... that we have not advanced to the homepage (we could also look for an error message such as "User not found"). We can't use login() because it will try to dismiss that modal after we login and throw an ElementNotFoundException or something like that, so we'll have to use clicks:

	def Test_Negative_Using_Clicks()
	lp = LoginPage()
	lp.username.type_text("invalidUname", "invalidPwd")
	lp.submit.click()
	lp.verify_on_page() # verify we're still on LOGIN page
	passed = true

view raw should02.rb hosted with ❤ by GitHub

Using Exceptions the Simple Way

Well that doesn't look too bad, but it was very-UI centric, and as our automation gets more complex, maintainability of these negative tests is going to become a factor. If all our positive testing is done using Flows, we've thrown it out the window and reverted to using page objects. Ok, let's actually try it with login() and just catch the exception:

	def Test_Negative_Using_Try_Catch()
	lp = LoginPage()
	try
	lp.login("invalidUname", "invalidPwd")
	catch ElementNotFound e
	if (e.error_msg == "Error: we are not on the Login page")
	passed = true

view raw should03.rb hosted with ❤ by GitHub

Using should_fail()

That works, but it's a lot of code. Let's hide and generalize that code in our should_fail() method:

	def Test_Negative_Using_Should_Fail()
	loginPage = LoginPage()
	# the arguments are:
	# the object you're calling,
	# the method you're calling,
	# any data you're passing to the method (possibly multiple values),
	# the expected error message
	passed = should_fail(loginPage,
	login,
	"invalidUname",
	"invalidPwd",
	"close_welcome_modal not found")

view raw should04.rb hosted with ❤ by GitHub

Ok, there's definitely some syntax to learn there, but not too much. Let's see how should_fail() works:

	# Here's the magic! I'm not sure how to do this in java,
	# the params would probably have to be an arary or something like that.
	# This is a simplified pseudocode version of how I've done it in
	# ruby in the past. There are a couple simple gotchas I've omitted.
	def should_fail(*args)
	target_object = args[0]
	method_to_call = args[1]
	arg_array = args[2..-2] # i love ruby
	expected_error = args[-1]
	try
	target_object.call(method_to_call, *arg_array)
	catch Exception e
	# if we get the error we expect, just return:
	if (e.to_string().match(expected_error))
	return true
	#if we get an error we DON'T expect, re-throw it with an explanation:
	else
	err="Failed: we expected the error: '{expected_error}' but got a different error: {e.to_string()}"
	throw(RuntimeError, e.to_string())
	# We were expecting an error, so if we did NOT get an error then that's bad,
	# so we throw an error with fact that we DIDN'T see any error...
	# Or you might want to return false instead of throwing an exception,
	# depends on your system design
	throw(RuntimeError, "Failed: we expected the error '{expected_error}' but the call '{target_object}.{method_to_call}' succeeded")

view raw should05.rb hosted with ❤ by GitHub

One thing you might notice is the hardcoding of the error messages. Those would be in some kind of dataStrings.rb file or something. The other is that I'm throwing around exceptions here quite a bit. This is not necessarily something you can shove into a pre-existing system that doesn't use exceptions very much, or doesn't use custom exception messages. All my wrapper click() and find() and type_text() and everything else methods throw exceptions with clear, relevant messages in order to make debugging easier.

Summary and Tying in Flows

So again, the whole point of this is so you can write your negative tests using Flow methods. Let's say in the example below that returning an order is a multi-step process. If we were to write this negative test using the page objects only similar to my Test_Negative_Using_Clicks() above, we would be reverting to a long test that was very tied to the UI. So by using positive testing of negative tests, we can stay user-centric and high-level, even if your task is complex. In the example below we're going to try to return an non-returnable item:

	# negative test - we're going to try to return a NON-RETURNABLE item
	class TestClass()
	def Test_Negative_Return()
	order_flow = OrderFlow()
	return_flow = ReturnFlow()
	test_data = get_test_data(...)
	order_flow.login(test_data.username, test_data.password)
	order_id = order_flow.order(test_data.order)
	order_flow.logout()
	return_flow.login(test_data.username, test_data.password)
	# try to return a NON-RETURNABLE item
	should_fail.(return_flow, return, order_id, "Error: item is not returnable")
	passed = true

view raw should06.rb hosted with ❤ by GitHub

The last thing I'll say is the error I'm looking for here is very high-level. Sometimes it's difficult or tedious to make your Flow methods do this, so maybe in this example, we would have tried to click a button as part of the return process but it wasn't there or was disabled, etc. In that case, we'd just have to catch the more generic "... does not exist" exception, and this does tie us to the UI. The usefulness of this system, like all automation, depends on the context of the system you're testing.