Software Testing Principles

1. A necessary part of a test case is a definition of the expected output or result.
2. A programmer should avoid attempting to test his or her own program.
3. A programming organization should not test its own programs.
4. Thoroughly inspect the results of each test.
5. Test cases must be written for input conditions that are invalid and unexpected, as well as for those that are valid and expected.
6. Examining a program to see if it does not do what it is supposed to do is only half the battle; the other half is seeing whether the program does what it is not supposed to do.
7. Avoid throwaway test cases unless the program is truly a throwaway program.
8. Do not plan a testing effort under the tacit assumption that no errors will be found.
9. The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section.
10. Testing is an extremely creative and intellectually challenging task.

Data Reference Errors

1. Does a referenced variable have a value that is unset or uninitialized?
• This probably is the most frequent programming error; it occurs in a wide variety of circumstances. For each reference to a data item (variable, array element, field in a structure), attempt to “prove” informally that the item has a value at that point.
2. For all array references, is each subscript value within the defined bounds of the corresponding dimension?
3. For all array references, does each subscript have an integer value?
• This is not necessarily an error in all languages, but it is a dangerous practice.
4. For all references through pointer or reference variables, is the referenced memory currently allocated?
• This is known as the “dangling reference” problem. It occurs in situations where the lifetime of a pointer is greater than the lifetime of the referenced memory. One situation occurs where a pointer references a local variable within a procedure, the pointer value is assigned to an output parameter or a global variable, the procedure returns (freeing the referenced location), and later the program attempts to use the pointer value. In a manner similar to checking for the prior errors, try to prove informally that, in each reference using a pointer variable, the reference memory exists.
5. When a memory area has alias names with differing attributes, does the data value in this area have the correct attributes when referenced via one of these names?
• Situations to look for are the use of the EQUIVALENCE statement in FORTRAN, and the REDEFINES clause in COBOL. As an example, a FORTRAN program contains a real variable A and an integer variable B; both are made aliases for the same memory area by using an EQUIVALENCE statement. If the program stores a value into A and then references variable B, an error is likely present since the machine would use the floating-point bit representation in the memory area as an integer.
6. Does a variable’s value have a type or attribute other than what the compiler expects?
• This situation might occur where a C, C++, or COBOL program reads a record into memory and references it by using a structure, but the physical representation of the record differs from the structure definition.
7. Are there any explicit or implicit addressing problems if, on the machine being used, the units of memory allocation are smaller than the units of memory addressability?
• For instance, in some environments, fixed-length bit strings do not necessarily begin on byte boundaries, but addresses only point to byte boundaries. If a program computes the address of a bit string and later refers to the string through this address, the wrong memory location may be referenced. This situation also could occur when passing a bit-string argument to a subroutine.
8. If pointer or reference variables are used, does the referenced memory location have the attributes the compiler expects?
• An example of such an error is where a C++ pointer upon which a data structure is based is assigned the address of a different data structure.
9. If a data structure is referenced in multiple procedures or subroutines, is the structure defined identically in each procedure?
10. When indexing into a string, are the limits of the string off-by-one errors in indexing operations or in subscript references to arrays?
11. For object-oriented languages, are all inheritance requirements met in the implementing class?

Data-Declaration Errors

1. Have all variables been explicitly declared?
• A failure to do so is not necessarily an error, but it is a common source of trouble. For instance, if a program subroutine receives an array parameter, and fails to define the parameter as an array (as in a DIMENSION statement, for example), a reference to the array (such as C=A (I)) is interpreted as a function call, leading to the machine’s attempting to execute the array as a program. Also, if a variable is not explicitly declared in an inner procedure or block, is it understood that the variable is shared with the enclosing block?
2. If all attributes of a variable are not explicitly stated in the declaration, are the defaults well understood?
• For instance, the default attributes received in Java are often a source of surprise.
3. Where a variable is initialized in a declarative statement, is it properly initialized?
• In many languages, initialization of arrays and strings is somewhat complicated and, hence, error prone.
4. Is each variable assigned the correct length and datatype?
5. Is the initialization of a variable consistent with its memory type?
• For instance, if a variable in a FORTRAN subroutine needs to be reinitialized each time the subroutine is called, it must be initialized with an assignment statement rather than a DATA statement.
6. Are there any variables with similar names (VOLT and VOLTS, for example)?
• This is not necessarily an error, but it should be seen as a warning that the names may have been confused somewhere within the program.

Computation Errors

1. Are there any computations using variables having inconsistent (such as nonarithmetic) datatypes?
2. Are there any mixed-mode computations?
• An example is the addition of a floating-point variable to an integer variable. Such occurrences are not necessarily errors, but they should be explored carefully to ensure that the language’s conversion rules are understood. Consider the following Java snippet showing the rounding error that can occur when working with integers:
• int x = 1;
• int y = 2;
• int z = 0;
• z = x/y;
• System.out.println ("z = " + z);
• OUTPUT:
• z = 0
3. Are there any computations using variables having the same datatype but different lengths?
4. Is the datatype of the target variable of an assignment smaller than the datatype or result of the right-hand expression?
5. Is an overflow or underflow expression possible during the computation of an expression?
• That is, the end result may appear to have valid value, but an intermediate result might be too big or too small for the programming language’s datatypes.
6. Is it possible for the divisor in a division operation to be zero?
7. If the underlying machine represents variables in base-2 form, are there any sequences of the resulting inaccuracy?
• That is, 10 Å~ 0.1 is rarely equal to 1.0 on a binary machine.
8. Where applicable, can the value of a variable go outside the meaningful range?
• For example, statements assigning a value to the variable PROBABILITY might be checked to ensure that the assigned value will always be positive and not greaterthan 1.0.9. For expressions containing more than one operator, are the assumptions about the order of evaluation and precedence of operators correct?
9. Are there any invalid uses of integer arithmetic, particularly divisions?
• For instance, if i is an integer variable, whether the expression 2*i/2 == i depends on whether i has an odd or an even value and whether the multiplication or division is performed first.

Control-Flow Errors

1. If the program contains a multiway branch such as a computed GO TO, can the index variable ever exceed the number of branch possibilities?
• For example, in the statement GO TO (200,300,400), i will i always have the value of 1,2, or 3?2. Will every loop eventually terminate?
• Devise an informal proof or argument showing that each loop will terminate.
2. Will the program, module, or subroutine eventually terminate?
3. Is it possible that, because of the conditions upon entry, a loop will never execute? If so, does this represent an oversight?
• For instance, if you had the following loops headed by the following statements:for (i==x ; i<=z; i++) {...}while (NOTFOUND) {...}what happens if NOTFOUND is initially false or if x is greater than z?
4. For a loop controlled by both iteration and a Boolean condition (a searching loop, for example) what are the consequences of loop fall-through?
• For example, for the psuedo-code loop headed by DO I=1 to TABLESIZE WHILE (NOTFOUND) what happens if NOTFOUND never becomes false?
5. Are there any off-by-one errors, such as one too many or too few iterations?
• This is a common error in zero-based loops. You will often forget to count “0” as a number. For example, if you want to create Java code for a loop that counted to 10, the following would be wrong, as it counts to 11:for (int i=0; i<=10;i++) {System.out.println(i);}Correct, the loop is iterated 10 times:for (int i=0; i <=9;i++) {System.out.println(i);
6. If the language contains a concept of statement groups or code blocks (e.g., do-while or {...}), is there an explicit while for each group and do the do’s correspond to their appropriate groups? Or is there a closing bracket for each open bracket? Most modern compilers will complain of such mismatches.
7. Are there any nonexhaustive decisions? For instance, if an input parameter’s expected values are 1, 2, or 3, does the logic assume that it must be 3 if it is not 1 or 2? If so, is the assumption valid?