Testing Overview and Black-Box Testing Techniques
Software testing is an important technique for assessing the quality of a software product.
In this chapter, we will explain the following:
• the basics of software testing, a verification and validation practice, throughout
the entire software development lifecycle
• the two basic techniques of software testing, black-box testing and white-box
• six types of testing that involve both black- and white-box techniques.
• strategies for writing fewer test cases and still finding as many faults as possible
• using a template for writing repeatable, defined test cases
1 Introduction to Testing
Software testing is the process of analyzing a software item to detect the differences
between existing and required conditions (that is, bugs) and to evaluate the features of
the software item. Software testing is an activity that should be done throughout
the whole development process .
Software testing is one of the “verification and validation,” or V&V, software practices.
Some other V&V practices, such as inspections and pair programming, will be discussed
throughout this book. Verification (the first V) is the process of evaluating a system or
component to determine whether the products of a given development phase satisfy the
conditions imposed at the start of that phase. Verification activities include testing
and reviews. For example, in the software for the Monopoly game, we can verify that
two players cannot own the same house. Validation is the process of evaluating a system
or component during or at the end of the development process to determine whether it
satisfies specified requirements. At the end of development validation (the second V)
activities are used to evaluate whether the features that have been built into the software
satisfy the customer requirements and are traceable to customer requirements. For
example, we validate that when a player lands on “Free Parking,” they get all the money
that was collected. Boehm has informally defined verification and validation as
Verification: Are we building the product right?
Through verification, we make sure the product behaves the way we want it to. For
example, on the left in Figure 1, there was a problem because the specification said that
players should collect $200 if they land on or pass Go. Apparently a programmer
implemented this requirement as if the player had to pass Go to collect. A test case in
which the player landed on Go revealed this error.
Validation: Are we building the right product?
Through validation, we check to make sure that somewhere in the process a mistake
hasn’t been made such that the product build is not what the customer asked for;
validation always involves comparison against requirements. For example, on the right
Are we building the product right?
Are we building the right product?
Both of Boehm’s informal definitions use the term “right.” But what is “right”? In
software we need to have some kind of standard or specification to measure against so
that we can identify correct results from incorrect results. Let’s think about how the
incorrect results might originate. The following terms with their associated definitions are helpful for understanding these concepts:
o Mistake – a human action that produces an incorrect result.
o Fault [or Defect] – an incorrect step, process, or data definition in a program.
o Failure – the inability of a system or component to perform its required function
within the specified performance requirement.
o Error – the difference between a computed, observed, or measured value or
condition and the true, specified, or theoretically correct value or condition.
o Specification – a document that specifies in a complete, precise, verifiable
manner, the requirements, design, behavior, or other characteristic of a system or
component, and often the procedures for determining whether these provisions
have been satisfied.
A mistake committed by a person becomes a fault (or defect) in a software artifact, such
as the specification, design, or code. This fault, unless caught, propagates as a defect in
the executable code. When a defective piece of code is executed, the fault may become a
visible anomaly (a variance from the specification or desired behavior) and a failure is
“I know this game has money and
players and “Go” – but this is not the
game I wanted.”
“I landed on “Go” but didn’t get my
observed. Otherwise, the fault remains latent. Testing can reveal failures, but it is the
faults that must be found and removed ; finding a fault (the cause of a failure) can be
time consuming and unpredictable. Error is a measure of just how incorrect the results
A purpose of testing is
to cause failures in order to make faults visible so that the faults can be fixed and not
be delivered in the code that goes to customers. Another purpose of testing is to assess
the overall quality level of the code. For example, a test team may determine a project
with too many high-severity defects should be sent back to development for additional
work to improve the quality before the testing effort should continue. Or, the
management may have a policy that no product can ship if testing is continuing to reveal
specification or desired
A programmer makes a
The mistake manifests
itself as a fault1 [or
defect] in the program.
A failure is observed if
the fault [or defect] is
made visible. Other
faults remain latent in
the code until they are
observed (if ever).
1.1 The Economics of Software Testing
In software development, there are costs associated with testing our programs. We need
to write out test plan and our test cases, we need to set up the proper equipment, we need
to systematically execute the test cases, we need to follow up on problems that are
identified, and we need to remove most of the faults we find. Actually, sometimes we
can find low-priority faults in our code and decide that it is too expensive to fix the fault
1 The IEEE does not define defect however, the term defect is considered to be synonymous with fault.
because of the need to redesign, recode, or otherwise remove the fault. These faults can
remain latent in the product through a follow-on release or perhaps forever.
For faults that are not discovered and removed before the software has been shipped,
there are costs. Some of these costs are monetary, and some could be significant in less
tangible ways. Customers can lose faith in our business and can get very angry. They can
also lose a great deal of money if their system goes down because of our defects. (Think
of the effect on a grocery store that can’t check out the shoppers because of its “down”
point-of-sale system.) And, software development organizations have to spend a great
deal of money to obtain specific information about customer problems and to find and fix
the cause of their failures. Sometimes, programmers have to travel to customer locations
to work directly on the problem. These trips are costly to the development organization,
and the customers might not be overly cheerful to work with when the programmer
arrives. When we think about how expensive it is to test, we must also consider how
expensive it is to not test – including these intangible costs as well as the more obvious
We also need to consider the relative risk associated with a failure depending upon the
type of project we work on. Quality is much more important for safety- or missioncritical
software, like aviation software, than it is for video games. Therefore, when we
balance the cost of testing versus the cost of software failures, we will test aviation
software more than we will test video games. As a matter of fact, safety-critical software
can spend as much as three to five times as much on testing as all other software
engineering steps combined !
To minimize the costs associated with testing and with software failures, a goal of testing
must be to uncover as many defects as possible with as little testing as possible. In other
words, we want to write test cases that have a high likelihood of uncovering the faults
that are the most likely to be observed as a failure in normal use. It is simply impossible
to test every possible input-output combination of the system; there are simply too many
permutations and combinations. As testers, we need to consider the economics of testing
and strive to write test cases that will uncover as many faults in as few test cases as
possible. In this chapter, we provide you with disciplined strategies for creating efficient
sets of test cases – those that will find more faults with less effort and time.
1.2 The Basics of Software Testing
There are two basic classes of software testing, black box testing and white box testing.
For now, you just need to understand the very basic difference between the two classes,
clarified by the definitions below :
o Black box testing (also called functional testing) is testing that ignores the
internal mechanism of a system or component and focuses solely on the outputs
generated in response to selected inputs and execution conditions.
o White box testing (also called structural testing and glass box testing) is testing
that takes into account the internal mechanism of a system or component.
The classes of testing are denoted by colors to depict the opacity of the testers of the code.
With black box testing, the software tester does not (or should not) have access to the
source code itself. The code is considered to be a “big black box” to the tester who can’t
see inside the box. The tester knows only that information can be input into to the black
box, and the black box will send something back out. Based on the requirements
knowledge, the tester knows what to expect the black box to send out and tests to make
sure the black box sends out what it’s supposed to send out. Alternatively, white box
testing focuses on the internal structure of the software code. The white box tester (most
often the developer of the code) knows what the code looks like and writes test cases by
executing methods with certain parameters. In the language of V&V, black box testing is
often used for validation (are we building the right software?) and white box testing is
often used for verification (are we building the software right?). This chapter focuses on
black box testing.
All software testing is done with executable code. To do so, it might be necessary to
create scaffolding code. Scaffolding is defined as computer programs and data files built
to support software development and testing but not intended to be included in the final
product . Scaffolding code is code that simulates the functions of components that
don’t exist yet and allow the program to execute . Scaffolding code involves the
creation of stubs and test drivers. Stubs are modules that simulate components that aren’t
written yet, formally defined as a computer program statement substituting for the body
of a software module that is or will be defined elsewhere . For example, you might
write a skeleton of a method with just the method signature and a hard-coded but valid
return value. Test drivers are defined as a software module used to involve a module
under test and often, provide test inputs, controls, and monitor execution and report test
results . Test drivers simulate the calling components (e.g. hard-coded method calls)
and perhaps the entire environment under which the component is to be tested .
Another concept is mock objects. Mock objects are temporary substitutes for domain
code that emulates the real code. For example, if the program is to interface with a
database, you might not want to wait for the database to be fully designed and created
before you write and test a partial program. You can create a mock object of the database
that the program can use temporarily. The interface of the mock object and the real
object would be the same. The implementation of the object would mature from a dummy implementation to an actual database.