International Agile Tester Foundation

Steen Lerche-Jensen

3 Agile Testing Methods, Techniques, and Tools

3.1 Agile Testing Methods

Agile or not, there are testing practices on every project or team which can be used to secure quality products. In addition, principles from the traditional testing are used a lot within the Agile testing approach. “Testing Early” is one of them, where a well-known Agile approach includes writing tests in advance to express proper behavior, focusing on early defect prevention, detection, and removal, and ensuring that the right test types are run at the right time and as part of the right test level. Let us have a look at some of the most used Agile testing approaches.


Figure: Test-Driven Development

3.1.1 Test-Driven Development, Acceptance Test-Driven Development, and Behavior-Driven Development

First, we will look at Test-Driven Development, often called TDD, but we will also look at Acceptance Test-Driven Development (ATDD), and Behavior-Driven Development (BDD). All three complementary techniques are in use among Agile Teams to carry out testing across the various test levels. All three techniques represent the traditional testing principle of “Testing Early”, since the tests are defined before the code is written.

Test-Driven Development (TDD)

Test-driven development (TDD) is used to develop code guided by automated test cases. The process for test-driven development is as shown in the figure:


Figure: Test-Driven Development. Key: See Tests Fail, Write Code, Run Tests, Refactor, Add Tests...

  • Add a test that captures the programmer’s concept of the desired functioning of a small piece of code
  • See the test fail. Run the test, which should fail since the code doesn’t exist
  • Write the code and run the test in a tight loop until the test passes
  • Refactor the code after the test is passed, re-running the test to ensure it continues to pass against the refactored code
  • Repeat this process for the next small piece of code, running the previous tests as well as the added tests.

Often this is at unit level and is code-focused, but can also be used at the integration or system levels. TDD became popular through Extreme Programming, but is also used in other Agile methodologies and also used by traditional testing projects. Using TDD makes it easy for the developers to focus on clearly-defined expected results. Normally these tests are automated and are used in Continuous Integration.

Acceptance Test-Driven Development (ATDD)

Acceptance Test-Driven Development is based on communication between the business customers, the developers, and the testers, and it defines acceptance criteria and tests during the creation of User Stories. Here are some key elements for ATDD:

  • Define acceptance criteria and tests early in development
  • Collaborate so every stakeholder understands behavior
  • Process (more about the process later):
    • Define tests for intended behavior
    • Create automated acceptance tests
    • Program intended behavior
    • Run automated acceptance tests
  • Create reusable regression tests for Continuous Integration
  • Test data and service layers
  • Test system and acceptance level in appropriate environments
    • Identify and quickly resolve defects
    • Validate feature behavior
    • Measure acceptance criteria
    • Deliver to external testing teams.

Jennitta Andrea has described ATDD as [Andrea, 2010]:
the practice of expressing functional story requirements as concrete examples or expectations prior to story development. During story development, a collaborative workflow occurs in which: examples are written and then automated; granular automated unit tests are developed; and the system code is written and integrated with the rest of the running, tested software. The story is “done”—deemed ready for exploratory and other testing—when these scope-defining automated checks pass.
Here some details regarding some of the points:

Acceptance criteria and "Testing Early"

The team agrees on a list of criteria, which must be met before a product increment, often a "User Story" is considered "done", and we can say that this defines acceptance criteria and tests during the creation of User Stories as early as possible.
One of the software testing principles says “Start Testing Early” in the software development lifecycle. This also applies to Agile projects. A lot of misunderstanding can be avoided through the lifecycle if we manage to get good and understandable “done criteria” as early as possible.
Also, remember Testing Early is not only about making use of acceptance criteria, it is also about review of existing documentation and other kinds of test basis, to make sure it has an acceptable content and structure for the team to work with. Good reviews can clean out a lot of misunderstanding up front.

Reusable tests for regression testing

Acceptance test-driven development creates reusable tests for regression testing. Regression testing is a type of software testing that seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes, such as enhancements, patches or configuration changes, have been made to them.
The aim of regression testing is to ensure that changes such as those mentioned above have not introduced new faults. One of the main reasons for regression testing is to determine whether a change in one part of the software affects other parts of the software. In Agile development, this is extremely important because we change the code in every Sprint. Therefore, we need to make sure that code that worked in a previous Sprint still works after new implementations. Regression testing is an integral part of Agile development methods and it is a common goal to automate as many of these tests as possible.

Continuous Integration process

Specific tools support creation and execution of such tests, often within the Continuous Integration process.
Continuous Integration is a software development practice where members of a team integrate their work frequently; usually each person integrates at least daily – leading to multiple integrations per day. An automated build (including test) to detect integration errors as quickly as possible verifies each integration. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

Continuous Integration

  • Fix regression and build bugs immediately
  • For efficiency, also automate test data generation and loading, build deployment, environment management, output comparison.

Example of Continuous Integration (CI), showing Source Control, Initiate CI Process, Build, Test, Testing, Report, Development, Commit.

Test data and service layer tools

These tools can connect to data and service layers of the application, which allows tests to be executed at the system or acceptance level.
ATDD allows quick resolution of defects and validation of feature behavior. It helps determine if the DONE criteria (acceptance criteria) are met for the feature.

Behavior-Driven Development (BDD)

BDD is similar in many ways to TDD except that the word “Test” is replaced with the word “Behavior”. Its purpose is to help the team to focus on testing the code based on the expected behavior of the software. BDD is usually done in very English-like language, which helps the Domain Experts to understand the implementation. Some key points:

  • Behavior-driven development is a black-box approach
  • Focus on expected behavior:
    • Developer creates a test for the class under development
    • Tests should make sense to stakeholders (including testers)
    • Clarify where defect lies (code, User Story, or test)
  • Helps to define test cases for developer based on tester/ stakeholder collaboration.

Specific behavior-driven development frameworks can be used to define acceptance criteria based on the given/ when/ then format:

  • Given some initial context,
  • When an event occurs,
  • Then ensure certain outcomes.

Here an example on a story that is described using the BDD method:

Story: Book returns go to stock

In order to keep track of book stock

As a book store owner

I want to add books back to stock when they are returned

 

Scenario 1: Refunded books should be returned to stock

Given a customer previously bought a book from me

And I currently have three books left in stock

When he returns the book for a refund

Then I should have four books in stock

 

Scenario 2: Replaced books should be returned to stock

Given that, a customer buys a softcase book

And I have two softcase books in stock

And three hardcase in stock.

When he returns the softcase for a replacement in hardcases,

Then I should have three softcase in stock

And two hardcase in stock

From these requirements, the behavior-driven development framework generates code that can be used by developers to create test cases. Behavior-driven development helps the developer collaborate with other stakeholders, including testers, to define accurate unit tests focused on business needs. There are several different examples of BDD software tools in use in projects today, for different platforms and programming languages.

3.1.2 The Test Pyramid

A software system may be tested at different levels. Often we show this as the Test Pyramid:


Figure: The Test Pyramid. Key: Unit, Integration, System, Acceptance.

  • Test pyramid emphasizes lower-level tests over upper-level tests
  • Automate unit and integration tests through API
  • Automate system and acceptance tests through GUI
  • Test pyramid concept is early QA and testing principle in action.

3.1.3 Testing Quadrants, Test Levels, and Testing Types

Testing quadrants align the test levels with the appropriate test types in the Agile methodology.

  • The testing quadrants associate test levels, types, goals, and focus
    • Cover important dynamic types and levels
    • Differentiate, describe types
  • Any test shown in any quadrant can occur in each iteration.

Agile Testing Quadrant as defined by Brian Marick. For Key, see earlier.

When asking how to find out which testing types should be used to test your Feature and Stories in Agile projects, it is useful to examine whether the tests are:

  • Business-facing, or
  • Technology-facing.

A business-facing test is one you could describe to a business/ Domain Expert. Here you are talking to the users of the system, using user terms and language: "If a book is sold out in your online book store, does the system automatically set up a message if a potential buyer searches for the book: “Book is sold out”, or will the search result not show the book in the search list?"
In the testing quadrants, tests can be business/ user-facing or technology/ developer-facing.
Some tests:

  • Support the work done by the Agile Team and confirm software behavior.
  • Other tests can verify the product.

Tests can be fully manual, fully automated, a combination of manual and automated, or manual but supported by tools. The four quadrants are as follows [ISTQB syllabus]:

  • Quadrant Q1 is unit level, technology-facing, and supports the developers. This quadrant contains unit tests (TDD).
  • Quadrant Q2 is system level, business-facing, and confirms product behavior. This quadrant contains functional tests, examples, story tests, user experience prototypes, and simulations. These tests check the acceptance criteria and can be manual or automated. They are often created during the User Story development and thus improve the quality of the stories (ATDD).
  • Quadrant Q3 is system or user acceptance level, business-facing, and contains tests that critique the product, using realistic scenarios and data. This quadrant contains exploratory testing, scenarios, process flows, usability testing, user acceptance testing, alpha testing, and beta testing. These tests are often manual and are user-oriented.
  • Quadrant Q4 is system or operational acceptance level, technology-facing, and contains tests that critique the product. This quadrant contains performance, load, stress, and scalability tests, security tests, maintainability, memory management, compatibility and interoperability, data migration, infrastructure, and recovery testing. These tests are often automated.

Tests from each of the quadrants can be used in any given iteration. For more information about the Agile testing quadrant, visit Brian Marick's blog [Brian Marick].

3.1.4 The Role of a Tester

If we take the Agile method Scrum as an example, we will now examine the role of “Tester” in the Agile lifecycle. We will look at the following:

  • Teamwork
  • Sprint Zero
  • Integration
  • Test planning
  • Agile testing practices

Teamwork

Agile is all about teamwork, often known in the Agile methods as the Whole-Team Approach. It combines three roles:

Figure: Teamwork. For Key, see earlier.

The three roles―Programmer, Domain Expert, Tester―represent the Team, but let us examine the role of “Tester”.

  • Teamwork is fundamental to Agile:
    • Cross functional: a group of people with different functional expertise working toward a common goal. Working together on test strategy, test planning, test specification, test execution, test evaluation, and test results reporting.
    • Self-organizing: a process where some form of global order or coordination arises out of the local interactions between the components of an initially disordered system. The team may consist only of developers, but ideally, there would be one or more testers and Domain Experts.
    • Collocated: Testers sit together with the developers and the Product Owner.
    • Collaborative: is working with others to do a task and to achieve shared goals. Testers collaborate with their team members, other teams, the stakeholders, the Product Owner, and the Scrum Master.
    • Empowered: Technical decisions regarding design and testing are made by the team as a whole (developers, testers, and Scrum Master), in collaboration with the Product Owner and other teams if needed.
    • Committed: An amusing anecdote is often told to show the strong commitment needed in Agile Teams:

A Pig and a Chicken are walking down the road.
The Chicken says: "Hey Pig, I was thinking we should open a restaurant!"
Pig replies: "Hm, maybe, what would we call it?"
The Chicken responds: "How about 'ham-n-eggs'?"
The Pig thinks for a moment and says: "No thanks. I'd be committed, but you'd only be involved!"

'I'd be committed, but you'd only be involved' said the Chicken to the Pig.

The tester is committed to question and evaluate the product’s behavior and characteristics with respect to the expectations and needs of the customers and users.

    • Transparent: used in science, engineering, business, the humanities and in a social context more generally, implies openness, communication, and accountability. Development and testing progress is visible on the Agile task board.
    • Credible: The tester must ensure the credibility of the strategy for testing, its implementation, and execution, otherwise the stakeholders will not trust the test results. This is often done by providing information to the stakeholders about the testing process.
    • Open to feedback: (from Latin retrospectare, "look back") generally means to take a look back at events that already have taken place. Feedback is an important aspect of being successful in any project, especially in Agile projects. Retrospectives allow teams to learn from successes and from failures.
    • Resilient: Testing must be able to respond to change, like all other activities in Agile projects.
  • Teamwork maximizes likelihood of successful testing

Sprint Zero

Sprint Zero is usually necessary because there might be things that need to be done before a Scrum project can start and it is the first iteration of the project where many preparation activities take place.
The tester collaborates with the team on the following activities during this iteration:

  • First iteration:
    • Identify scope
    • Divide User Stories into Sprints
    • Create system architecture
    • Plan, acquire, and install tools
    • Create initial test strategy for all test levels
    • Perform initial quality risk analysis
    • Define test metrics
    • Specify the definition of “done”
    • Define exit criteria
    • Create the task board
  • Sets the direction for testing throughout Sprints.

Tester role in Integration

In Agile projects, the objective is to:

  • Continuously deliver customer value:
    • In each Sprint
    • At the end of the project.
  • Continuous testing requires identifying all dependencies between functions and features
    • Integration strategy is helpful for this process
    • Important to identify all dependencies between underlying functions and features.

Test Planning

Since testing is fully integrated into the Agile Team, test planning should start during the release planning session and be updated during each Sprint. Planning will include the following tasks:

  • What: test scope, extent of testing, test and Sprint goals
  • Why: explain what’s in and out of scope
  • Who: assign testing of features and characteristics
  • Where: which test environment and what changes, data, tools, and configurations are needed for it (think ahead)
  • When: start date and frequency for test tasks
  • How: which test methods, techniques, tools, and data
  • Prerequisites: predecessor tasks, expertise, training
  • Dependencies: functions, code, system components, vendor, technology, tools, activities, tasks, teams, test types, test levels, and constraints
  • Project and quality risks: as described in Foundation and later
  • Priority: consider customer/ user importance, risks, dependencies
  • Time: duration and effort required to test, as described later
  • Populate task board: Sprint planning populates task board with tasks (1-2 days).

Agile Testing Practices

Here is a short list of some Agile practices, which may be useful for testers in a scrum team:

  • Pairing: Pair testing is a software development technique in which two team members work together at one keyboard to test the software application. One does the testing and the other analyzes or reviews the testing. This can be done between one tester and developer or business analyst or between two testers with both participants taking turns at driving the keyboard.
  • Incremental test design: Test cases and Test Charters are gradually built from User Stories and other test bases, starting with simple tests and moving toward more complex ones.
  • Mind mapping: A mind map is a diagram used to visually organize information. A mind map is often created around a single concept, drawn as an image in the center of a blank landscape page, to which associated representations of ideas such as images, words and parts of words are added. Major ideas are connected directly to the central concept, and other ideas branch out from those. Mind mapping is a useful tool when testing. For example, testers can use mind mapping to identify which test sessions to perform, to show test strategies, and to describe test data.

Figure: How to Create a Mindmap. BLUE: Nobody's perfect, Don't focus on perfection, Tidy up later, Let your ideas explode. ORANGE: Break the 'on the page' mentality, Your brain isn't confined to one page, So your Mindmaps shouldn't be either. VIOLET: Don't take sides, Embrace both sides of your brain, The Creative, The Analytical. RED: Use Color, Group your ideas and thoughts by using color. INDIGO: Share it, When you finish, share it with your Study Buddies, This will help you get a fresh perspective. GREEN: Go Deeper, Every node on the Mindmap could be its own Mindmap.

Use the promo code: atfacademy10 and get 10% discount for the International Agile Tester Foundation Certification