Agile Testing: History, Background, Principles

A Comprehensive Overview of the Groundbreaking Methodology

If you mention Agile Testing in a crowded room, you are likely to see heads nodding in general agreement. Ask those same people what the term means and you are likely to get answers that range from different semantic differences to polar opposite opinions on what the definition of Agile is.

Even experts disagree on their preferred way to test in an Agile environment, or what an “Agile” environment is. This ebook will focus on the origins of the term, the ideas that came together from the Agile Manifesto, how those ideas have played out over the years, and some lessons learned.

It was nearly two decades ago that seventeen leaders came together at the Snowbird Ski Resort to draft The Agile Manifesto. The manifesto represented a different way of thinking about software projects - a feedback-driven approach, driven by the people actually doing the work.

The authors of the Manifesto came from different places within software development; most of them were more technical than their peers in the “process improvement” community. The major groups represented include Schwaber and Sutherland, who co-created Scrum, which focuses on letting the team self-organize, but ship working software periodically. Extreme Programming (XP), which had advocates including Ron Jeffries, Robert Martin, and Kent Beck, initially eliminated all support roles, focusing only on programmer and customer. XP got the middle-people out of the way, broke the work down into small chunks called stories, and tested features to predict progress. Another popular method, the Dynamic Systems Development Method, or DSDM, treats time, quality and resources as fixed, and provides tools to flex the scope.

But it was Brian Marick, the only person at the gathering identified as a tester, who summed the ideas up: "The [skilled] software people have figured out how to build systems; please allow them to do it."

Many readers will be familiar with the “cover” of the Manifesto:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.

Fewer are aware of the twelve principles of the Manifesto. If the cover is the why, the principles are the “what”, including the idea of welcoming changing requirements, of shipping software frequently (at the time, measured in weeks to a couple of months), and that face-to-face conversation is the most effective way to convey information.

Shipping every few weeks creates a problem for testing. Pre-Agile, most projects had a “testing phase” that was also a “fixing phase.” The test process itself could take from a few weeks to a few months. Yet if software was supposed to ship frequently, testing would need to find a way to adapt and keep up. Agile-Testing, is therefore testing designed to enable teams to practice the values of the manifesto. If the manifesto says the more effective and efficient way of communicating information is face to face, that means rethinking heavyweight, documentation-focused handoffs.

An ideal agile testing process would do more than be a glove for the development fist; it would be integrated into the process itself. Testing ceases to be a separate activity and becomes a continuous part of the development cycle. In the nearly two decades since the manifesto was penned, a great number of attempts have been made to move toward that ideal. Here are a few, and the state of the industry today.

The first Extreme Programming (“XP”) project, in the late 1990’s, was Chrysler's Consolidation Compensation project, also known as C3. C3 had no user interface; the software read in files that contained hours and employee statistics, then ran a batch job to create another series of files to be sent to a print vendor to create physical checks or process direct deposits.

The C3 project had no external test group. Instead, programmers worked in pairs, creating unit tests as they went using Test Driven Development. The project also had automated acceptance tests, for a given type of employee and set of hours, generate this kind of check. This lack of a user interface was perfect for programmer-type testing, with clearly defined inputs, transformations, and expected results.

The popularity of Extreme Programming led to two popular ideas - that there would be no testing role, and that all tests should be automated. In Testing Extreme Programming, Lisa Crispin and Tip House actually have a chapter titled “Manual Tests';” the body of that chapter is single sentence that there are no manual tests in XP.

It’s easy to misunderstand that chapter. You can imagine it as an executive at a product demonstration, asking what happens if the customer uses a name that is identical to an existing user in the system. The product owner looks at the lead programmer, who says “Good question.” The product owner moves to set up the data, and the programmer says “Wait! No! That’s manual testing! You can’t do that! Let’s write down the scenario and I’ll let you know what happens at the next sprint review.”

Human exploration happens all the time, officially or unofficially. Teams that recognize this and try to get the most value for the least effort will have the best long-term results. Another major innovation that came from the XP world was Continuous Integration. Having a server check out, compile, and at least run unit tests automatically was a major innovation; getting alerts when unit tests failed tightened the feedback loop for bug fixing, in some cases from weeks to hours.

While XP provided the “technical how-to” for Agile Software Development, it was Scrum that won the brand wars and would emerge as the default standard for Agile Software Development. Scrum provides the container to do the work in (the “sprint”) along with some roles and rituals - but is surprisingly silent on how to do the work and especially how to test. In their 1990 book Wicked Problems, Righteous Solutions, DeGrace and Stahl include this in their description of Scrum. “After you have gotten them used to the idea that they are special, having been specifically chosen to do an important job, you further unsettle the team by saying that their job is to produce the system in, say, half the time and money and it must have twice the performance of other systems. Next, you say that how they do it is their business. Your business is to support them in getting resources. Then, you leave them alone.”

The ideas including automated testing, continuously checking, having no specific test role, and leaving the team to decide how to do the work, resulted in nearly two decades of experiments and innovations. While there is still no consensus today, a few strong ideas have emerged.

It was the context-driven software movement, invited to the Snowbird conference but not in attendance, that championed an alternative to “automate everything” - human exploration, or exploratory testing. Today Exploratory Testing, which involves test design, execution, reporting and learning feeding into each other continuously, is considered a staple of modern Agile-Testing.

An open debate you may hear today is the necessity of the tester role. Most organizations with a graphical user interface or complex business rules still employ testers although they may have fewer testers as a percentage, and expect other roles to perform test activities, which is known as whole-team testing. Many organizations find that while a tester role might not be needed, it can certainly be helpful.

Two of the more important innovations in Agile-Testing are the test automation pyramid, proposed by Mike Cohn, and the Agile-Testing Quadrants, published by Brian Marick and popularized by Janet Gregory and Lisa Crispin. Cohn’s pyramid suggests that teams should start with a large number of programmer tests, or unit-tests, having few tests between components, and a very small number of automated User Interface (UI) checks. This means more of the faster-running, less-brittle tests, and less of the slower-running, more brittle tests. It tends to create systems that have higher quality before getting to testing the User Interface, instead of “testing the bugs out” at the UI level.

The Testing Quadrants, published in the book Agile Testing, suggests that tests can exist to help the team move faster (“support the team”) or to assess the quality of the product. The value is that it points to several kinds of testing a team might consider. Even if they choose, for example, not to perform internationalization testing because the product is intended for an English-only audience.

Most of the discussion around Agile-Testing is about dynamic testing - running a piece of code or program find information. That approach means first creating errors and then removing them. Modern Agile Testers try to test and find defects as early as possible. Unit Tests can find or even prevent defects during the programming cycle, as can programmer/tester pairing. The collaboration, or blending, between dev and test roles that happens during pairing is part of the magic of agile-testing well.

Another common type of collaboration in agile-testing is having all roles involved in creating not just a “spec document”, but a set of examples. These examples can drive development and testing. At least three roles build this specification together, one small feature at a time. First is the developer, who wants to know how the software should behave so they can do a good job. Next is the tester who has knowledge of failure modes and common errors, and can come up with good example and edge cases. Finally, the product owner (PO) or analyst has responsibility for the feature and generally final decision making authority. PO’s tend to work at a broader level than technical staff, and need expert questions to dig into the weeds, to create a first-time system with higher quality prior to testing. That group, which George Dinwiddie graciously coined as The Three Amigos, is an example of the value of collaboration - of working together and whole-team testing.

It’s easy enough to break Agile-Testing down into three categories. Programmers create unit tests, perhaps called Micro-Tests or “Small Tests.” Testers and people engaged in testing test at lesat twice: Story-Tests, where a small feature or piece of functionality is explored, and regression testing, where the system is tested as a whole just prior to release. Some companies have managed to eliminate regression-testing and instead deliver a continuous flow of value. Doing that takes a lot more than testing skill. It takes the ability to isolate components, test them and deploy them at the individual level, notice and rollback production problems quickly. Those solutions are typically found in the worlds of DevOps and Continuous Delivery.

These ideas are evolving; Crispin and Gregory’s latest “definition” admits to this, and repeats the key themes outlined above. Their working summary of Agile Testing is that it includes:

Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting frequent delivery of value for our customers. Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding. The practices strengthen and support the idea of whole team responsibility for quality.

In the great system, Agile Testing can plug into a Continuous Delivery Pipeline, offering early and frequent information. Some of that runs automatically through code, others run through human exploration by refining ideas before they are committed to code. This combination finds important issues early and also provides insights that make a better product - the state of the practice of Agile Testing today.