Acceptance Testing


22
Jul 09

Pdf Assert

I’ve been working with the folks in Standard Life Investments recently, and I’ve been on an automated Fund Fact Sheets project.
It’s an interesting project, in that external vendors supply the system as per spec.

So, one of the things we’re doing is shipping Acceptance Tests as spec. One of the outputs of that spec is PDF reports.

I’ve initially spiked a PDF regression testing tool (like for like comparison) and some features that you can use for Acceptance Testing.
I then went away and wrote this TDD for them, but I’m interested in knowing if anyone else would find this useful.

If so, I can re-write the spike and release it. Add a comment if you would like to see this…


20
Feb 09

What Happens When The Acceptance Tests Pass?

When an Acceptance Tests passes it becomes a Regression Test.

That test, after passing, typically gets moved out of the Acceptance Test bucket into the Regression Test bucket.

The reason why this is important is that Acceptance Tests, whilst they are checked in, do not fail the automated build.  However, when a Regression Test fails, it does.


20
Feb 09

Who Should Run Acceptance Tests? And When Should They Be Run?

The tests are there for 2 main audiences: developers and the business.

Developers typically run Automated Acceptance Tests against the work that they are developing to verify that they’ve done the work that’s expected.

The business normally views Acceptance Test results to ascertain where the iteration is at any one time.
They also qualify as executable Acceptance Criteria for the business.


20
Feb 09

How do I write an Automated Acceptance Test?

Be particular.

A typical dialog would be:

Customer: “It doesn’t matter to this story we’re doing who the user is.”

Test Writer: “The Developers understand John A Smith as the Persona we use when it doesn’t matter.  He has a wife, 1 child, 1 infant, flies LHR to CDG, 1 way fixed economy.  Is that okay to make that assumption?”

Customer: “Yup, as long as the correct promo code is applied”

Test Writer: “What is the correct promo code?”

Customer: “£25 fixed discount, irrespective of spend”

Test Writer: “Are there any other codes needing applied for this story”

Customer: “Yes…”
etc.

This could transfer into code like this:

Persona john = new Persona(“JohnASmith”);
FareQuotePage fareQuotePage = new FareQuotePage();
fareQuotePage.enterPromoCodeFor(john);
Assert.assertEquals(“£350”, fareQuotePage.getFullPrice());
Assert.assertEquals(“£100”, fareQuotePage.getDiscountToBeApplied());
Assert.assertEquals(“£250”, fareQuotePage.getTotalWithDiscount());


20
Feb 09

Who Should Write an Acceptance Test?

In an ideal world Acceptance Tests should first and foremost be written by the Customer, or Customer Proxy with Test involvement.

In reality, depending upon the technical abilities and makeup within the team, developers may be needed to “bootstrap” the writing of tests in order for others to take over.

This is time well spent as it gives developers a very good familiarity with the story before it arrives.


20
Feb 09

Why Does Writing Automated Acceptance Tests Force Out Ambiguities?

For example, consider the following as a plain English Acceptance Test:

Somebody goes to the page, enters a discount voucher and gets the right amount off?

The format for writing Acceptance Tests is normally like this:

A [named user role] can [select/operate] [feature/function] so that [output] is [visible/complete/etc.]

Now, as writing the test requires real data, and real results it could transpire to this:

A web user, John A Smith (who is flying with his wife Jane, 1 child called Jack, 1 infant called Janet) chooses economy class, one way to Paris Charles De Gaul, can enter a £25 fixed price discount voucher so that the full fare of £350 is reduced to £250?

The latter is a test which has a yes or no answer and is not ambiguous.

There can be a shorthand employed within the test case if everyone who reads the tests understands what that shorthand is, like so:

A web user, John A Smith, can enter a £25 fixed price discount voucher so that the full fare of £350 is reduced to £250?

This implies that John A Smith is a Persona understood by all to be:

John A Smith
Passenger Choice:
2 Adults: John A Smith and Jane Smith,
1 Child: Jack Smith,
1 Infant: Janet Smith
Flight: Economy, 1 way, CDG

In fact, when writing a scenario that doesn’t really care about certain variables, the Persona can be re-used as an archetypal good guy, with the atomic family, thus reducing waste in data set-up costs.

They also become data sets that everyone can talk about and form a common language for people to talk round.


20
Feb 09

When Should Automated Acceptance Tests be written?

Automated Acceptance Tests should be written as part of story writing and estimating exercise, before any work has started on the story.

Some teams are comfortable to write the automated tests before estimating.

Others prefer a plain English description of the tests as being good enough for estimating and prefer to include the test writing effort as part of the story.

In either case, the tests typically get written before any work has begun on the story.

The effort of writing tests, using either approach, brings out any ambiguities within the story, and forces the right questions at the right time.


20
Feb 09

At What Level Should I Write an Acceptance Test?

Acceptance Tests are written in order to capture what is acceptable for the delivery of the software.
In the web world, an Acceptance Test typically would involve exercising the user interface and asserting suitable state change within other systems when required.
Acceptance Testing at this level has previously been considered relatively expensive.
However, the more recent application of an object oriented approach (as opposed to a procedural one) has been applied to the problem of page changes and has greatly reduced the brittleness of these kinds of tests.


20
Feb 09

What are Automated Acceptance Tests?

Automated Acceptance Tests are human readable documents that are executable.
Typically this takes the form of them running as part of an automated build within Continuous Integration (CruiseControl, Hudson etc)

They are derived from an Agile practice called Functional Testing (meaning to cover a  function which has yet to be developed).

There are typically 2 different states for these Functional Tests:-
Regression: these cover functionality already delivered and should pass when run.
Acceptance: these cover functionality which has yet to be developed and should fail.  These form the criteria acceptable to the customer.

Writing Automated Acceptance tests before any further work irons out exactly what has been agreed to be delivered and informs the design and development activities.


20
Feb 09

Approaches on Sensibly Dividing Up Buckets of Tests.

In a more advanced state, Acceptance and Regression tests are typically split into 4 buckets.
The first two cover the current iteration, the other buckets are beyond the current iteration scope.
The following diagram illustrates the life-cycle of Acceptance to Regression path.  This particular example is based on a 2 week iteration.

Typical Acceptance and Regression Test lifecycle - iteration

Typical Acceptance and Regression Test lifecycle - iteration

Basically, when a test passes, it moves from an Acceptance Test bucket within the iteration to a Regression Test bucket within the iteration.  Once the iteration finishes (hopefully!) all the Acceptance Tests have become Regression Tests, and can be moved into a larger bucket of tests.
Points to note
The previous Regression Test bucket, and end to end bucket can get very large, very quickly.
There needs to be continual intermittent effort to validate the tests in these areas.
Normally the following questions will be asked:
1)Do we still need the test?  Does the functionality still exist and needs to be covered?
2)Do we need to update the test to make it relevant?
3)Is there duplication?

However, note that removing tests in order to go quicker needs to be considered on a risk basis.