Lecture 19: Functional Testing

Today's Lecture

Congrats to two of our students
- Aaron Cummings
- Graham Roberts
- Paul Steinbrecher
Both competed in the ACM Programming Contest this past weekend
- Paul's team came in first (out of 12 teams) in the local competition and was ranked fourth overall (out of 53 teams) in the region. They solved 5 problems out of 9 problems total.
- Aaron and Graham's team came in seventh in the local competition and thirty-second overall in the region. They solved 3 problems.
Later today, I will be making the problem set available at this URL: http://www.cs.colorado.edu/~kena/acmpc/2006/

Program Verification
- A program is correct if it meets its requirements specification
Requirements Specs
- F(input) = output
- Functional Contract, should be as specific and precise as possible
Test Case
- Input, Documentation, and Expected Output
- Test Suite: a collection of test cases
Test Run
- Actual Output and Pass/Fail Grade
- Run each test case and record pass/fail; repeat until all tests pass

How do you pick test cases?
Two main approaches
- Functional testing (aka black box testing)
- Structural testing (aka white box testing)
Note: modern software testing has moved past these concepts but they are still useful as an introduction to the testing field

In functional testing, we test the functionality of the system without regard to its implementation
The system is, in a sense, a black box because we cannot look inside to see how it computes its output
We provide input and receive output

Functional testing is a strategy for helping a software engineer pick test cases
This is useful, since selecting test cases is a tricky problem
- A test suite should be “complete”…
  - with respect to the program's specification
  - but how many test cases do you need to be complete?
- A test suite should be “precise”…
  - no duplicate test cases
  - if a test suite takes too long to run, then it will get run less often (which increases the chance that a fault goes undetected)

Functional testing helps create test suites by providing a criterion for selecting test cases:
The requirements specification of a program lists functions that the program must perform
A test suite is complete when it tests every function
For each function, determine “categories” of input that a function should treat equivalently (as we did with the GCD function in Lecture 17)
- boundary conditions can be useful guides
- test both “typical” input and error conditions
- a test suite will need at least one test case for each category associated with each function

Identify functional categories in the requirements specification that broadly classifies functionality the program must perform
Example: IT System for a Car Dealer
- Car Database
  - including creation, deletion, and update of entries for each car owned by the dealership
- Employee Database
- Payroll
- Inventory Forecasting
- Report Generation
- Sorting

Identify specification items in the spec that correspond to functions the program must perform
Each specification item should be assignable to one of your functional categories
- This can be an iterative process, in which a specification item identifies a new functional category
Example
- The user shall be able to create a new entry in the car database. A unique id will automatically be assigned to this entry. The user shall then enter additional information about the new car in the fields provided. (Car Database)
- The user shall be able to see a list of cars sorted by the time a car has been in the inventory from longest to shortest (report generation, sorting)
- The system shall notify users by e-mail whenever a direct deposit into their bank account has occurred (payroll)

Identify categories (aka functional equivalence classes) for each specification item found in step 2 (similar to the GCD example in lecture 17)
Example
- Specification Item: The user shall be able to see a list of cars sorted by the time a car has been in the inventory from longest to shortest
- Categories:
  - List is requested on an empty car database
  - List is requested on a car database with a single entry
  - List is requested on a car database with many entries
  - List is requested on a car database that has been edited (entries added, then removed and/or updated)
  - etc. (You need to determine when enough is enough... for some items it may be possible to come up with a ton of categories; restrain yourself and pick only the most critical)

Determine test cases for each category
- You may only have one test case per category; however, its okay to pick more than one test case per category
- Be on the look out for boundary conditions; sometimes they are handled in the categories (i.e. zero, one, or many cars in database), sometimes they are not. In the latter case, you will then need more than one test case in the category to cover each boundary condition
- This means that in functional testing, “there is more than one way to skin a cat”; one person's categories may be another person's test case, may be another person's specification item. It all depends on the level of granularity that you set for each concept.
Identifying Test Cases
- List is requested on an empty car database: one test case (create empty database, invoke function)
- List is requested on a car database with a single entry: one test case (create empty database, add one car, invoke function)
- …

Eliminate redundant test cases
- You may have test cases that test different (categories of different) specification items using the same input
- If so, you can eliminate the duplicates and then annotate all effected specification items by saying something like “this category is tested by test case 2 of category 1 of specification item 10”.

Prioritize test cases
- You may not have the time or budget to test them all
- As such, give critical test cases higher priority…
- …while test cases that test obscure or uncommon errors can be given lower priority
You now have your final test suite!