Today's Lecture

• Introduce the concept of program verification
  • specifications
  • terminology
  • debugging
  • testing

Program Verification

• Program verification is the process of demonstrating that a particular program meets its specification
  • If a program meets its specification it is considered “correct”

Program Correctness

• To repeat: a program is correct only when it meets (i.e. implements) its specification
• This statement can have contradictory implications:
  • Without a specification, a program can NEVER be correct.
  • Or the exact opposite, you could decide that since there is no specification, ANY behavior is (vacuously) correct
• Note: a program that meets its specification is not necessarily useful
  • If the specification is nonsense, then the program is nonsense.
  • Or if the specification contains an error then, while the program is still considered “correct”, its not meeting the needs of its users

Program Specifications

• You can view a program’s specification abstractly as a function that maps the program’s inputs to its expected outputs
  • F(input) = output
  • e.g. if you click on this button, a menu pops up
  • F(click on button) = menu pops up
• Remember that this way of thinking is “an abstraction”
  • The “real world” is much more complex. For instance, a program may be “correct” on a machine with 64MB of memory, but fail on a machine with 32MB of memory
    • This type of error is related to the “non-functional” requirements of a system
    • In this class, we will be focusing on “functional” requirements only
  • Furthermore, the example above is fairly low level, you may want to write specifications at a higher level of abstraction

Testing Terminology

Discussion

• A failure occurs only if a fault occurs, and a fault occurs only if an error exists
• Note: not all faults are detected, because you may need to execute a specific portion of a program for the failure to appear…
  • …and it may be impossible to execute all “states” of a program.
• Recall that Fred Brooks in No Silver Bullet talked about the number of states associated with software systems and how they can have at least one order of magnitude more states than their associated hardware systems

An Example

• Imagine a program with the following statement:

if (x < y)

• but imagine that the programmer made a mistake and should have typed:

if (x <= y)

Creating Correct Programs (1 of 3)

• Strategy One: Error Prevention
  • Employ techniques to avoid errors in the first place
    • Software Reuse
    • Design before you Code
    • Smart Programming Environments
      • auto-balance, syntax coloring, snippets, …

Creating Correct Programs (2 of 3)

• Strategy Two: Debugging
  • The process of discovering and eliminating faults
  • Tools can help with this
    • compiler warnings, source debuggers, …
  • Review can help as well
    • Code Inspections

Creating Correct Programs (3 of 3)

• Strategy Three: Testing
  • The process of discovering failures and locating the faults which cause them
  • Give input to a program and compare its generated output with the expected output
  • Testing involves creating test cases that “cover” (or rather “uncover”) different types of failures
  • Module and Integration Testing is done by developers (or QA), system testing is done by the customer!

Creating Test Cases

A test case consists of:

Test Run

Test cases are applied to a program during a test run. A test run consists of:

Test runs are typically supported by a “testing harness” or “test scaffolding”. This refers to the software that helps you perform (or sometimes automate) test runs

Testing Process

For each class of failure defined in the documentation For each test case in that class Apply the input and compare the output to the specification Record results Fix discovered problems Repeat until all test cases pass

Creating Test Cases

• How do you pick test cases?
  • We will look at two strategies for doing this
    • Black Box Testing
    • White Box Testing
• For now, think of trying to pick “categories” of input that test the same thing
• Its impossible to “exhaustively” test a program…
• …but if your categories contain values that all test the same thing, you can get by with using just a single value from each category

Example

int GreatestCommonDivisor(int x, int y)

1. x=6 y=9, returns 3, tests common case
2. x=2 y=4, returns 2, tests when x is the GCD
3. x=3 y=5, returns 1, tests two primes
4. x=9 y=0, returns ?, tests zero input value
5. x=-3 y=9, returns ?, tests negative input value

To test exhaustively is impossible (both parameters can take on an infinite number of values) but with 5 categories identified, we can get by with only 5 test cases!

Coming Up Next

• Lecture 18: Requirements Specifications
• Lab 3 Due