And why should we incur the cost of making and running Regression Tests?
It’s a new year, and a new gig.
Nice bunch of folks, and they’re a fair way down the line of adopting Agile Management Practices.
They also recognise that, whilst they are adopting a fair few technical practices, they nod their heads to the fact that they could potentially do better.
So whilst I and my wife were travelling around Indonesia my colleague arrives at the team and does a “testing MOT” (the client’s words – nice phrasiology!).
So my colleague spots something that is normally done as a technical practice in Agile teams but this team hasn’t adopted yet: running a core set of regression tests before checking in, and writing automated Acceptance tests that wil eventually turn into Regression.
He makes a report recommending paying back the test debt that the manual testers are feeling right now, and then moving on to greener pastures where you can catch automatic Acceptance tests before the code is written.
S0, I land (and go from 30 oC to 5 oC) and promptly go back to work the next day. I get there, get the brief from my colleague (all sounds pretty darn reasonable to me!) and get cracking on immediately doing a technical spike (best way to begin – code doesn’t lie) .
So, its feasible [the approach is outlined in another article From Regression to Acceptance - How To Evolve Sensibly] to start to write Regression, and later, Acceptance tests over the front end. By this I mean a few core front end tests, with fixed data that exercises the top 3 tiers of the app that they are working on, and a good few lower risk tests that will run in their Continuous Integration environment.
And I’m happy that they can: often when you arrive after the fact testing is more difficult, and often you can only do tracerbullet-like tests.
I report my progress back to the team and stakeholders with a fair level of confidence that I can help them.
Then a reaction I didn’t expect: two questions from one of the main stakeholders (and echoed by the team):
- What’s exactly the value [monetary gain] of Acceptance Testing?
- Why should we incur the cost of making and running Regression Tests? (meaning the testing team normally picks this up, and this is just extra lifting for us)
I didn’t have an immediate answer. I’d expected they already had seen the value after their ‘testing MOT’…
And its a strange thing: every team, bar none, I’ve ever worked on has just done it, no questions.
So, now I’m thinking, I know why you do it, but I’m darned if I know how to attach a value to the activity.
Okay, the easy bit, the why
- Defects past dev are more expensive to fix
- Tight feedback circles give the developer more confidence that the code they are developing is right
- Unit testing doesn’t necessarily test the collaboration effectively
- Manual testing will eventually come to a ‘hard limit’ due to eventually having more regression to run than people capable of doing so
- A machine is good at repetition, humans are crap at it
- Executable Acceptance Tests will give you a clear progress indicator as to how much the folks have delivered
So, I’ve, in all likelihood, missed a good few. However, even with that lot I couldn’t attach any monetary value in the why. That is often a thing after the fact.
Time for an anecdote I think:
A few years back I was working with a team (who, co-incidentally, was the smartest team I’ve worked with)
They had previously been doing front end, all the way through the system, testing using DbUnit to set up the data to a known good state.
A few of the folks were starting to get really pissed with the weight of these front end testing and decided to forego that activity on the new project they started on.
This project was a pay-per-transaction based web app for a big ol’ mobile handset manufacturer. They worked their ass off delivering this app, and had a few late nights getting it right.
The customer was over the moon that the system was delivered on time, with all the new features. Job well done, and we didn’t have to do any of those pesky bothersome tests. Halfway through the first week the Product Manager wants a tiny (life or death) change to layout as quickly as possible.
It was done. No-one gave another thought to it.
Two days later:
“The new site we’ve launched has transactions at 10% of our old one”
“Maybe they haven’t put it live yet their end?”
“I doubt it, someone look into it please?”
A javascript error on the page before the transaction page. Hmmm, that 10% must’ve had javascript turned off.
Value of testing found out after the fact? Well, somewhere in the upper tens of thousands of pounds in lost revenue, and the dev effort spent to fix it.
Now there’s a value attached. But that couldn’t have been predicted.
That’s the essence of my quandry.
I don’t want to be a smart ass and say “what’s the cost of not doing it? or catching it late?” as they cannot relate to something that may not happen…