Lecture 20: Structural Testing

Today's Lecture

Discuss Structural Testing
- Terminology
- Techniques
- Examples

Structural Testing

Supplies another criteria to answer the question: “How many test cases are enough?”
Recall that functional testing’s criteria was “Test all functions”
Structural Testing’s criteria is “Test all code”
- Structural Testing is also known as white box testing, because now we look at a program’s source code to help create test cases

Control Flow Graphs (CFGs)

The basis for structural testing is a data structure known as the CFG, or control flow graph.
Control flow graphs capture the various ways in which a program can execute
- A node in a CFG represents a program statement
- An edge in the CFG represents the ability for a program to flow from its current statement to the statement at the other end of the edge
  - If an edge is associated with a conditional, it is labeled with the conditional's value, either true (T) or false (F)

Control Flow Example

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Types of Code Coverage

Statement Coverage
- Every statement is executed at least once
Edge Coverage
- Every edge is traversed at least once
Condition Coverage
- For binary logical operators (&&, ||), the individual components are evaluated in every possible combination of true and false
Relational Coverage
- For relational operators (<, >, <=, >=) the equal condition is treated as a separate branch
Path Coverage
- Every possible path is executed at least once

White-Box Testing Criteria

Statement Coverage
- Execute each statement at least once
Pick test case and plot its path through the CFG
Continue to pick test cases until all statements are covered

Statement Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=20, Y=10

Statement Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=20, Y=10

Statement Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=10

Statement Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=10

Statement Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=10

Statement Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=10

Statement Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=30

Statement Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=30
Output: 50

Statement Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=30
Output: 50

Note: We just don't care about the non-executable
statements in the code above. Lines 1, 2, 3, 8, 11,
13, and 15 are simply not significant with respect
to the control flow graph.

Statement Coverage: Test Case 2

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=20, Y=30

Statement Coverage: Achieved!

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Test Case 1: X=20, Y=10
Test Case 2: X=20, Y=20

White-Box Testing Criteria

Edge Coverage
- Traverse each edge at least once
- Pick test case and plot its path through CFG
- Keep picking test cases until all edges are covered
Also known as Branch Coverage
- We must traverse each conditional (such as an if statement) along its true and false edge
  - without regard to short circuit operators
  - Short circuit boolean operators are ones which do not necessarily test each of their operands. For instance, given the following statement:
```
if (A && B) then
```
    the truth value of B will only be checked if the truth value of A is true. If A is false, there is no need to check the truth value of B since the entire expression is guaranteed to be false. Likewise given the following statement:
```
if (A || B) then
```
    the truth value of B will only be checked if the truth value of A is false. If A is true, there is no need to check the truth value of B since the entire expression is guaranteed to be true.

Edge Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=20, Y=10

Edge Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=20, Y=10

Edge Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=10

Edge Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=10

Edge Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=10

Edge Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=10

Edge Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=30

Edge Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=30
Output: 50

Edge Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=10, Y=30
Output: 50

Note: We still cover statements like we did before,
but now the focus is on the edges that we hit.

Edge Coverage: Test Case 2

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=15, Y=30

Edge Coverage: Achieved!

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Test Case 1: X=20, Y=10
Test Case 2: X=15, Y=30

Note: We missed one edge but since it has to do
with a short circuit operator, edge coverage doesn't care.

White-Box Testing Criteria

Condition Coverage
- Traverse each edge at least once BUT…
  - pay attention to complex conditionals (especially short circuit operators) and make sure that all possible combinations of true and false are tested
- Pick test case and plot its path through CFG
- Keep picking test cases until all edges are covered

Condition Coverage: Test Case 1

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=20, Y=10

Condition Coverage: Test Case 2

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=5, Y=30

Condition Coverage: Test Case 3

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Input: X=21, Y=10

Condition Coverage: Achieved!

 1  function P return INTEGER
 2  begin
 3      X, Y: INTEGER;
 4      READ(X); READ(Y);
 5      while (X > 10) loop
 6          X := X - 10;
 7          exit when X = 10;
 8      end loop;
 9      if (Y < 20 and then X mod 2 = 0) then
10          Y := Y + 20;
11      else
12          Y := Y - 20;
13      end if;
14      return 2 * X + Y;
15  end P;

Test Case 1: X=20, Y=10
Test Case 2: X=05, Y=30
Test Case 3: X=21, Y=10

Relational Coverage

Relational Coverage
- This is a form of edge coverage in which any relational operator (<, >, <=, and >=) has its equal condition treated as a separate branch

So, “if (x < y)” should be treated as

if (x < y) then
    …
else if (x == y) then
    …
else if (x > y) then
    …
end

Relational Coverage, continued

Relational coverage is thus a stronger form of edge coverage
It is saying that for each conditional, you should have at least three test cases
- x < y
- x > y
- x == y
Combine this approach with condition coverage and you have the strongest form of edge coverage possible

Path Coverage

Path Coverage
- Traverse each path through a program at least once
Problem
- Way too many paths, even in simple programs
Approach
- Use heuristics
  - e.g. for each loop take loop zero, one, and multiple times

Path Coverage Example

Question

How many paths does the following program fragment have?

a << cin; b = 0;
while (a > 0) {
    a--; b++;
}
if (b > 5) {
    printf("b > 5");
} else {
    printf("b <= 5");
}

Answer
- For any particular value of a, there is one path through the program
- BUT since a is entered by the user, there are an infinite number of possible paths!

Coming Up Next

Lecture 21: Extreme Programming (will be presented by the TA)
Quiz 8: Available after this lecture. Don't forget to take it!!!
Homework 8 Assigned

Software Methods and Tools

Lecture 20: Structural Testing

Today's Lecture

Structural Testing

Control Flow Graphs (CFGs)

Control Flow Example

Types of Code Coverage

White-Box Testing Criteria

Statement Coverage: Test Case 1

Statement Coverage: Test Case 1

Statement Coverage: Test Case 1

Statement Coverage: Test Case 1

Statement Coverage: Test Case 1

Statement Coverage: Test Case 1

Statement Coverage: Test Case 1

Statement Coverage: Test Case 1

Statement Coverage: Test Case 1

Statement Coverage: Test Case 2

Statement Coverage: Achieved!

White-Box Testing Criteria

Edge Coverage: Test Case 1

Edge Coverage: Test Case 1

Edge Coverage: Test Case 1

Edge Coverage: Test Case 1

Edge Coverage: Test Case 1

Edge Coverage: Test Case 1

Edge Coverage: Test Case 1

Edge Coverage: Test Case 1

Edge Coverage: Test Case 1

Edge Coverage: Test Case 2

Edge Coverage: Achieved!

White-Box Testing Criteria

Condition Coverage: Test Case 1

Condition Coverage: Test Case 2

Condition Coverage: Test Case 3

Condition Coverage: Achieved!

Relational Coverage

Relational Coverage, continued

Path Coverage

Path Coverage Example

Coming Up Next