Automation Framework: Wrap it up!

TL;DR you should wrap Selenium, Appium, etc. with a class of your own, even if it's not adding much value at this moment.  

The simplest reason is that if today you're calling myButton.click() and you're using Selenium's click() method in a hundred different places in scripts and libraries, what if you want to change the behavior of click()?  Why would anyone ever want to do that?  There are a few different reasons including waiting before clicking, retrying after a failed click, or my favorite: logging.

Maybe you want to go into debug mode and log every click.  If you're using Selenium's click() method, you may be out of luck.  If you're using your own click method, which previously did nothing but call Selenium's click(), just add a logging line:

Automation Framework: Specifying Element Type such as Button/Link/Dropdown Considered Harmful

I have worked in 5 frameworks over the past 12 years and although I keep seeing it, I am still unconvinced that knowing the exact type of your element adds value.  If I want to click an Element, I call .click().  Knowing that it's a Button or Link certainly makes no difference because the interfaces to those element types are the same.  Knowing whether it's a Button or a Dropdown I still argue make no difference, my_element.click() will do what I want it to do.  It's like duck typing in Ruby.  In highly complex systems, having very robust typing may be helpful in avoiding mistakes and making the code easier to read, but most UI automation just isn't that complex.

If I want to know exactly what type of element I'm dealing with, I have to write a bunch of classes such as

class Button (webElement):
  def click():
     self.click()
  ...


class Dropdown (webElement):
  def getOptions():
    select = new Select(self.driver, self.by)
    return select.getOptions()
  ...


Or maybe we could group them into related groups (but how related is related?) such as

class

class Clickable (webElement):
  def click():
     self.click()

class Button (Clickable):
  pass

class Link (Clickable):
  pass
...

This way at least we're saving a little redundancy with shared functionality, but still, what's the point?  I'm all for writing a wrapper for whatever automation tool you're using (Selenium), but without being specific about types.  When the rubber meets the road - i.e. when you're testing your test script - you'll find out if you've done it right.  What I mean is if you think you have a Dropdown and you call getOptions() but you actually have a Link, it'll fail.  You don't have to write the extra code that allows you to call that object a Button or Dropdown object vs an Element object.  If you had that extra code, maybe it would've failed 10 seconds earlier when you tried to instantiate a Dropdown object but passed a button web element to it.

This leads to my next point:  the greater specificity you have in what type of Element you have, the LESS flexible your test is.  If you're testing your application's logic, you shouldn't care as much about clicking a button or a link.  What you should care about is what the user cares about: logging in, placing an order, viewing a document... some business value.  If you care about testing the UI and whether that web element is a button or a link then ok, maybe you should be specific, but I've never worked anywhere that this was the case.

This lack of flexibility will rear its head when your app undergoes a refactor.  When all those buttons turn to links as you move from an older-style UI to a newer one, your tests all break. Even if they don't, you now have all these elements called orderButton or showMapButton which are actually links in the UI.  If you had been using generic Element objects, there would have been no code to change.

QA Basics: Bug Priority vs Severity by Example

Severity is how adversely the bug affects the software or the user's experience.  Priority is simply the order in which a bug will be fixed.  This is most easily explained by example.

• High severity, high priority: Not being able to log in.  Major functionality is lost and the impact is critical.
• Low severity, low priority: A typo on some screen that users rarely use.  No functionality lost, low impact.
• Low severity, high priority: At Nike, the "Swoosh" is considered sacred. A backwards or upside down Swoosh is seen extremely rarely and if it's an accident, people over there LOSE THEIR MINDS. If a page incorrectly showed an upside down Swoosh, it would be low severity because the app still functions just fine, but it would be high priority because they would want it fixed immediately. 
• High severity, low priority: If your app has a feature with a button that, when clicked, crashes the whole app, that would be high severity.  If that feature is almost never used by any of your customers, it would be low priority.  If that sounds odd, believe me, there are millions of these kinds of bugs out there in the wild.  Known fatal bugs that no one has plans to fix.

Typically the QA Engineer will set the severity (and sometimes priority) when they log the bug, and then it will be reconsidered and possibly updated when the bug is triaged. Triage is a process, often weekly, in which a representative of QA, Dev, and Product get together and go through new bugs and decide if and when to fix them.  Priority may be assigned during triage or by the product team independently of others.

QA Basics: How to Write a Defect Report

Generally, a bug/defect report should be self-contained and include all information required to reproduce and triage the bug.

Clarity and reproducibility are key.  The better bugs you write, the happier the devs will be.  Happy devs write happy code.  Happy code means less bugs!

The key components of a bug/defect report are:

• Title
• A concise summary, don't include reproduction steps (at some point you'll be tempted to)
• Description
• A discussion of what is wrong, possibly how it affects the user.  If you are familiar with the issue or the code, give that information.
• If there is a history with this issue or this type of issue, include it here.
• Environment
• version or build #
• platform or server
• browser (Chrome, FireFox, etc.)
• data such as user info (could also just be in Steps to Reproduce)
• Can include other things (on mobile you could include things like connection type of WiFi or Cellular)
• Steps to reproduce
• Actual result
• Expected result
• Evidence
• screenshots, logs, stack traces, etc.
• for search ability: copy/paste key parts of logs, error msgs, and the top several lines of stack traces.  Screenshots of logs or err msgs are fine but ALSO paste the text in the description or a comment
• Priority
• How urgent should we fix this?
• Severity
• How severe are the consequences of this bug when it does happen?

I like to format my Steps to Reproduce as shown below, to allow for a discussion after each step (numbers = actions, dashes = how the app behaves).

Most companies don't have strict formats, this is just my personal style guide (just as dev teams have style guides for coding).

1.login to www.xyz.com as user01 password abcd1234

2.put any item > $100 in the cart

-cart shows the item but not the tax

3.go to settings page, then go back to the cart page

-cart now shows item AND the tax

I've included multiple actions in step 3.  Personally I think this is ok as long as it's unmistakably clear.

Priority vs severity:

See this post.

Email: The easy way to remember how to write a test plan

Team,

Something you’ll probably have to do at some point is to write a test plan.  There is an IEEE standard for writing test plans but no one follows it (maybe the military).  Test plans vary wildly from company to company.  Too small and it won’t be enough information, too big and no one will ever read it.

It can be hard to remember what goes in a test plan but it’s a common practice and interview question, so I think of it like this:

• Who
• Who are the stakeholders
• Stakeholders can be loosely defined as “people with an interest in the project” such as project managers, product managers, key customers, your boss, etc.
• If needed, write a RACI chart
• Who will be doing the testing
• Who is on the team, do we have enough people, do we have the right people
• What
• What will we test (what is the scope of testing)
• Include a list of high-level areas we will test.  How specific you get will depend on the project. 
• What will we NOT test
• Include a list of what you will NOT test.  Sounds weird but this is very important to include.  This will include things like
• the customer’s servers
• security
• When
• Do we have a delivery date or a list of milestones (phases)?
• If we need to deliver by date X, development must deliver to us by date Y
• Where
• Environmental factors
• “we need a server with version 1.2 as well as version 1.4”
• How
• A discussion of tools we’ll use, testing types we’ll do (and maybe processes)
• We’ll use the API
• We’ll use Selenium
• We’ll do load testing
• We’ll test directly in the database
• We’ll use processes X, Y, Z 
• Why
• …doesn’t really work in this who/what/when way of thinking

On top of all of this, a few things to include are

• Risks
• the dates are tight here, any shift in development will push us back
• some environment goes down a lot or is out of our control (maybe it relies on the client or someone 3rd party)
• Dependencies
• we need credentials for X to complete testing
• we need better specifications for Y part of the system
• Results
• How will you measure your success
• How will you deliver that data

Email: Auto-commit is a big scary monster that will steal your wallet (database updates)

Hey all,

There was an “unfortunate incident” the other day where one of our “developers” did something I like to call “forgetting the WHERE clause” during a DB update in UAT.  Let’s try to avoid this.  This is just from my experience – Nick or anyone else is welcome to chime in here for best practices

1.Turn off auto-commit

• in DbVisualizer, go to settings by hitting Cmnd+,  (this is a handy osx standard)
• click the database tab
• choose MySQL
• click Transaction
• uncheck Auto Commit
• uncheck the boxes under “Ask when Auto Commit is OFF” unless you want a popup every time
• click Apply
• click OK

2.Always do a SELECT first, then modify it

don’t just look at the data and start writing your UPDATE or DELETE statement

write your select statement and validate that ONLY the rows you want to change will be updated:

SELECT * FROM fulfiller_account_settings

WHERE fulfiller_id = 96576

AND fkey='CARRIER'

---only 1 row, good!

Now change SELECT * FROM to UPDATE

And add your SET clause:

UPDATE fulfiller_account_settings

SET fvalue = 'CanadaPost'

WHERE fulfiller_id = 96576

AND fkey='CARRIER'

17:26:19  [UPDATE - 1 rows, 0.160 secs]  Command processed

... 1 statement(s) executed, 1 rows affected, exec/fetch time: 0.160/0.000 sec  [1 successful, 0 errors]

---only 1 row, good!

Now click the little “database with a floppy disk” icon in the toolbar to commit.  To see if auto-commit is on or off, look in the area between the input/output windows (see green arrows in the screenshot).

A couple final notes:

1.If you do an update or delete WITHOUT auto-commit: when you do a select, the data WILL reflect your changes.  The rest of the world will NOT see those changes and you have effectively done nothing until you commit, but to you it will look like the changes are made

2.You can do multiple transactions before doing a commit (someone correct me if I’m wrong)

3.You cannot roll back after a commit
4.If you have not committed and want to roll-back any changes in the buffer, hit the “database with a red dot” button to the Right of the commit button (see green arrow in the screenshot).