International Agile Tester Foundation

Steen Lerche-Jensen

3.2 Assessing Quality Risks and Estimating Test Effort

Risk-Based Testing (RBT) is known from both traditional and Agile projects. Risk-Based Testing is a type of software testing that functions as an organizational principle used to prioritize the tests of features and functions in software, based on the risk of failure, the function of their importance and likelihood or impact of failure. Moreover, it attempts to estimate the effort required to minimize those risks to an acceptable level or remove them totally.


Risk-Based Testing

To mitigate the risk we use test design techniques. Often we use the same kind of techniques known from traditional testing, but due to the short iterations and the often many changes in Agile development lifecycles, some adaptations of those techniques are required.
In short, Risk-Based Testing means:

  • Reduce product (quality) risk to an acceptable level
  • Lightweight techniques can be used:
    • Identify quality risks
    • Assess level of risk
    • Estimate test effort
    • Mitigate risks through test design, implementation, and execution
  • Short iterations and fast change have implications.

3.2.1 Assessing Quality Risks in Agile Projects

Prioritizing, allocation and selection of the test condition is one of the biggest challenges within both traditional and Agile testing. However, let us have a look at some of the content in assessing Product/ Quality risks:

  • Select, allocate, and prioritize test conditions to maximize effectiveness and efficiency
  • Product/ Quality risk analysis supports this process
    • Risk: a possible negative outcome
    • Level of risk: based on likelihood and impact
    • Product/ Quality risks: potential problems with product quality
    • Project risks: potential problems for project success
  • Agile quality risk analysis is performed:
    • At a high level during release planning by business stakeholders
    • At a detailed level during iteration planning by the whole team
  • In each iteration, the tester designs, implements, and executes tests for the risks.

Example of a Risk Matrix showing Threat Level, based on Likelihood (Low, Medium, High) and Impact (Low, Medium, High)

Examples of Product/ Quality risks for a system include:

  • Incorrect calculations in invoice (a functional risk related to accuracy)
  • Slow response to user online payment transaction (a non-functional risk related to efficiency and response time)
  • Difficulty in understanding an error message (a non-functional risk related to usability and understandability).

Example: Difficulty in understanding an error message

Product/ Quality Risks

  • Product Risks (also called Quality Risks) include all features and attributes of the product that can affect customer, user, stakeholder satisfaction
    • Incorrect calculations (functional)
    • Slow response time (non-functional performance risk)
    • Confusing interface (non-functional usability risk)
  • Risk analysis prioritizes tasks and guides the sizing of the tasks
    • High risks require extensive testing, come earlier, and involve more story points
    • Low risks receive cursory testing, come later, and involve fewer story points
  • Risk-based prioritization also includes Sprint backlog items.

Product Risk/ Quality Risk analysis process in an Agile project

  1. Gather the Agile Team members together, including the tester(s)
  2. List all the backlog items for the current iteration (e.g., on a task board)
  3. Identify the quality risks associated with each item, considering all relevant quality characteristics
  4. Assess each identified risk, which includes two activities: categorizing the risk and determining its level of risk based on the impact and the likelihood of defects
  5. Determine the extent of testing proportional to the level of risk
  6. Select the appropriate test technique(s) to mitigate each risk, based on the risk, the level of risk, and the relevant quality characteristic.

The tester and the team

The tester in the team (can be anyone) then designs, implements, and executes tests to mitigate the risks. This means looking at the whole picture to satisfy the customer, User and other stakeholders:

  • Features
  • Behaviors
  • Quality characteristics, and
  • Attributes.

Change can happen during the release or Sprint, therefore, the team should remain aware of additional information that may change the set of risks and/or the level of risk associated with known quality risks. From time to time adjustment of the product risk analysis will take place. Adjustments include:

  • Identifying new risks
  • Re-assessing the level of existing risks, and
  • Evaluating the effectiveness of risk mitigation activities.

Review up front is an example on how to mitigate product risk before test execution starts.

3.2.2 Estimating Testing Effort Based on Content and Risk

Just as the Agile Team estimates the functionality development in a User Story during release planning, the team also need to address the testing effort:

  1. Test strategy is defined during release planning
  2. During iteration planning, User Stories are estimated (e.g., via Planning Poker® in story points)
  3. Story points give implementation effort
  4. Risk level should influence story points
  5. Planning Poker® can be used to reach consensus, involve whole team, and avoid missing anything
  6. Reliable estimation, including testing, is necessary for smooth work pace and meaningful velocity.

Let us have a look at some techniques.

Agile estimation

All the entries within the Scrum Product Backlog have to be estimated to allow the Scrum Product Owner to prioritize the entries and to plan releases – This includes estimating the test effort.

Planning Poker®/ Scrum Poker

One commonly used method during the estimation process is to play Planning Poker® (also called Scrum Poker). When using Planning Poker®, influences between the participants are minimized and therefore a more accurate estimation result is produced.
In order to play Planning Poker® the following is needed:

  1. The list of features to be estimated
  2. Decks of numbered cards.

Example of number sequence cards for estimating features.

The Scrum Framework itself does not prescribe a single way for the Agile Teams to estimate their work. However within the Scrum Framework the estimation is not normally done in terms of time – a more abstracted metric to quantify effort is used. Common estimating methods include numeric sizing (1 through 10), tee-shirt sizes (XS, S, M, L, XL, XXL, XXXL) or the Fibonacci sequence (0, 1, 2, 3, 5, 8, 13, 21, 34, etc.).
What is important is that the team shares a common understanding of the scale it uses, so that every member of the team is comfortable with it.
A typical deck has cards showing the Fibonacci sequence including a zero: 0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89; other similar progressions are also possible. The reason for using the Fibonacci sequence is to reflect the uncertainty in estimating larger items. A high estimate usually means that the story is not well understood in detail or should be broken down into multiple smaller stories. Smaller stories can be estimated in greater detail. It would be a waste of time to discuss if it is 19, 20 or 25; the story is simply (too) big.
The game is then played in the following steps:

  1. The Scrum Product Owner presents the story to be estimated. The Scrum Team asks questions and the Scrum Product Owner explains in more detail. If many stories have to be estimated a time-constraint (e.g. only one minute for explanation) might be set as well. If the time-constraint is reached and the Scrum Team does not understand the story it is a sign that the story has to be re-written.
  2. Each member of the Scrum Team privately chooses the card representing the estimation.
  3. After everyone has chosen a card, all selections are revealed.
  4. People with high and low estimates are allowed to explain their estimate.
  5. Estimation starts again until a consensus is found.
  6. This game is repeated until all stories are estimated.

In addition, the following should be taken into account when estimating test effort:

  1. Extensive: run large number of tests, both broad and deep, combine and vary interesting conditions, use all relevant techniques with strong coverage criteria
  2. Broad: run medium number of tests, exercise many different interesting conditions, use most relevant techniques with medium coverage criteria
  3. Cursory: run small number of tests, sample most interesting conditions, use efficient techniques with weak coverage criteria
  4. Opportunity: leverage other tests or activities to test 1-2 interesting conditions, investing very little time and effort, using reactive techniques especially
  5. Report bugs only: allocate only a small amount of extra time to report and manage these accidental bugs.

Use the promo code: atfacademy10 and get 10% discount for the International Agile Tester Foundation Certification