Software testing. Lesson 5

1.

Software Testing

2.

Content
1. Definitions
2. Software Testing Methods
3. Software Testing Levels
4. Software Testing Types
5. Quality and Its Criteria
2
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

3.

PREAMBLE
Application testing was seriously taken in ninetieth in the U.S. Those years
were mad ones for High Tech industry. Mass application of internet, wide use
of quick-develop-technologies caused the creation of huge programming
systems at that time. And the complexity has been growing day after day.
Consequently the quantity of potential problems has been growing as
well.
It is difficult to take own place in the market selling really poor,
incomprehensible programs with a lot of defects. And nowadays most of
companies, producing software, pay more and more attention on their
testing.
3
Definitions
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

4.

SOFTWARE TESTING
Software Testing – is a process of software analysis and defect detecting.
Actions of defect detecting are directed to defining as many as possible
defects, which should be fixed.
IEEE Standard Glossary of Software Engineering Terminology says that
testing is «The process of exercising software to verify that it satisfies
specified requirements and to detect errors».
Why is testing a process?
Because testing is a planned and well-ordered activity. This moment is
important because under time shorted conditions well planned and
systematic approach leads to defects detecting more rapidly than poorly
planned testing activity.
4
Definitions
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

5.

DEFECT
Defect (bug) – non-conformance to requirements or functional specification. It is something
that does not correspond to valid Customer’s expectations that are assumed but may be
not described in product requirements. A defect may have a place in requirements or any
other documentation, in design or architecture. A bug can be result of incorrect
environment, configuration or data.
Expected Result – the behavior of the system (software product) we expect to see in result
of our actions (inputs). In other words – this is correct behavior of the software product.
If the actual behavior of the program we see does not correspond to that one which we
expect (not the same), such behavior may be considered as wrong and we may say that
we have found the defect.
5
Definitions
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

6.

TEST CASE, PLAN, BUILD
Test Case – a set of test inputs, execution conditions and expected results
developed for a particular objective, such as to exercise a particular program
path or to verify compliance with a specific requirement.
Test Plan – is a part of the overall project documentation describing what,
when, how and who will be involved in the testing process. This is basic
document that describes a list of tested components, quality criteria and
risks of testing, resources and graphs of testing, testing strategy and testing
types.
Build – is an intermediate version of a software product which is released
by developers for testing.
6
Definitions
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

7.

WORK PRODUCTS TO BE TESTED
Work Products that can be tested:
Programs at installing, launching
and operating.
Program’s code without launching.
Product prototype.
Product documentation:
Project documentation:
o Product requirements;
o Product architecture, product
o Functional specifications;
design.
o On-line help;
o Project plan, Test plan.
o Installation guide and user
o Test cases.
manual.
7
Definitions
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

8.

SOFTWARE TESTING APPROACHES
Static Testing is a process, which is associated with analysis of software.
Static testing provides for testing (verifying) of any work product, e.g. code,
requirements documents, functional specification, architecture, design
documents, etc. Static testing is one of the most effective ways of defects
detecting in the early stages of product process. Actually static testing is
what can be made for defects detecting without running the code.
Dynamic Testing is a testing activity providing for software operating.
Dynamic testing can not avoid running the code. Better to say, dynamic
testing consists of launching the program, running all functional modules
and comparing the product’s behavior with expected one using user
interface.
8
Approaches
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

9.

SOFTWARE TESTING METHODS
Black box testing. Testing software based on functional and business requirements at
launching and operating it without knowledge of the internal structure or program source
code. A tester tests a product so as an end-user would work with it at launching and
operating it. The purpose of this method is to check the proper work of all functions, to
verify whether all functions correspond to functional requirements.
Advantages:
Disadvantages:
• Tester does not need programming
knowledge.
• Tests are done from the point of view of
the client (end user), not the
designer/coder.
• Test cases can be designed as soon as the
specifications (or requirements) are
complete.
Tests can be redundant if the coder has
already run the same test cases.
Test cases are difficult to design - very
difficult when coding is done ahead of
specifications/design documentation.
Testing every possible input stream is
unrealistic because it would take a
inordinate amount of time.
9
Methods
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

10.

SOFTWARE TESTING METHODS
White box testing (glass box testing). Tester uses his understanding of source code and
access the code to develop and execute test cases. This method tests the architecture of
the system. It tests the design and programming that goes into building system. White box
testing is usually applied when application is not entirely assembled, but it is necessary
to check each of the components, modules, procedures and sub-functions. White box
testing is close interrelated with unit testing, which is often performed by developer who
understands the logic of code.
Advantages:
Disadvantages:
• Some hidden problems can be revealed • We need good technical skills to perform
easily.
such tests.
• Bug diagnosis can be done quickly.
• It is difficult to imitate end user behavior
• We can «run» such tests even if we have
– some «typical» bugs can be missed.
no executable code yet.
10
Methods
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

11.

BLACK BOX VS WHITE BOX TESTING
Synonyms for black-box include: behavioral, functional, opaque-box, and closed-box.
Synonyms for white-box include: structural, glass-box and clear-box.
Generally black box testing will begin early in the software development i.e. in
requirement gathering phase itself. But for white box testing approach one has to wait
for the designing has to complete.
We can use black testing strategy almost any size either it may be small or large. But
white box testing will be effective only for small lines of codes or piece of codes.
In white box testing we can not test Performance of the application. But in Black box
testing we can do it.
11
Methods
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

12.

SOFTWARE TESTING LEVELS
Time
Unit testing
Integration testing
System testing
System integration testing
Level of
specificity
Regression testing
Acceptance testing
Alpha testing
Beta testing
12
Levels
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

13.

TIME
Unit Testing. This level of testing allows to perform the testing of separate module of
the system. It may be a testing even of any particular part of the code (class).
Integration Testing. Testing of combined parts of an application to determine if they
function together correctly. Also, interactions between applications of big system can
be checked within this type of testing. In this case this testing is known as Crossproduct testing. Usually it is performed after unit and functional testing of modules.
System Testing. Checking the operation of the system in a whole. This testing checks
as functional as non-functional requirements.
Integration
Testing
Unit
Testing
System
Testing
13
Levels
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

14.

TIME
System Integration Testing verifies that a system is integrated to any
external or third-party systems defined in the system requirements.
Testing the integration of systems and packages; testing interfaces to
external organizations (e.g. Electronic Data Interchange, Internet).
14
Levels
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

15.

LEVEL OF SPECIFICITY
Acceptance Testing: Formal testing with respect to user needs,
requirements, and business processes conducted to determine whether
or not a system satisfies the acceptance criteria and to enable the user,
customers or other authorized entity to determine whether or not to
accept the system.
Regression Testing: Testing of a previously tested program following
modification to ensure that defects have not been introduced or
uncovered in unchanged areas of the software, as a result of the
changes made. It is performed when the software or its environment is
changed.
15
Levels
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

16.

LEVEL OF SPECIFICITY
Alpha Testing: Simulated or actual operational testing by potential
users/customers or an independent test team at the developers’ site,
but outside the development organization. Alpha testing is often
employed for off-the-shelf software as a form of internal acceptance
testing.
Beta Testing: Operational testing by potential and/or existing
users/customers at an external site not otherwise involved with the
developers, to determine whether or not a component or system
satisfies the user/customer needs and fits within the business
processes. Beta testing is often employed as a form of external
acceptance testing for off-the-shelf software in order to acquire
feedback from the market.
Levels
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.
16

17.

FUNCTIONAL TESTING
Functional Testing is the process of attempting to find discrepancies
between the program and its functional requirements or specifications.
Functional testing may be performed manually (manual testing) or
using specific tools (automated testing).
Some quotations:
«Testing can be used to show the presence of bugs but never their
absence». (E. Dijkstra)
«Testing is the process of executing a program with the intention of
finding errors». (G. Myers, The Art of Software Testing, Wiley, 1979)
17
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

18.

FUNCTIONAL TESTING
The goals are:
Defining defects in software product and its documenting.
Determine if software meets its requirements as defined in the
Software Requirements Specification (SRS) and other relevant
documents.
Take an objective decision about possibility of software product
delivery to customer; the decision should be documented in test
result report.
18
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

19.

FUNCTIONAL TESTING LEVELS
Smoke Test
It is the first and the shortest test, which checks the basic
functionality of software product. This test takes approximately 14 hours depending on program complexity, by results of which lead
tester takes a decision if to continue further testing or not.
Critical Path Test
It is a basic type of test when functions significant for end-user are checked on regular
working basis. In view of this type of testing, most of requirements to the product are
verified as a rule.
Extended Test
It is a profound test, when non-typical system’s behavior is checked. When logically tricky
scenarios are run, and some actions are performed which end-user may perform rarely.
19
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

20.

FUNCTIONAL TESTING LEVELS
New Feature Testing
Testing the new features that have been added since the previous
test cycle (since previous build).
20
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

21.

POSITIVE/NEGATIVE TESTING
Positive Testing is that testing which attempts to show that a given
module of an application does not do what it is supposed to do. We pass
absolutely correct data, execute application in absolutely correct
environment and conditions. If it fails – things are bad.
Negative Testing is that testing which attempts to show that the
module does something that it is not supposed to do. We «play tricks»
with application, i.e. try to divide by zero or so on. Application is
supposed to react correctly (i.e. to point to user mistakes but not to
fail).
21
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

22.

FUNCTIONAL TESTING: RECOMMENDATIONS
Make software fail, don’t just confirm that it works.
Focus on abnormal and stressful conditions, boundary testing.
Testing against specifications and outside specifications.
Assume the hardware may fail.
Do not plan tests assuming that no errors will be found.
Be skeptical of everything!
Involve Customers if you can, act Like a Customer if you can’t.
Emphasize testing of critical functions.
22
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

23.

NON-FUNCTIONAL TESTING
Non-functional testing refers to aspects of the software that may not
be related to a specific function or user action such as scalability or
security. Eg. How many people can log in at once? Non-functional
testing is also performed at all levels like functional testing.
Non-functional testing includes:
•Reliability testing
•Usability testing
•Efficiency testing
•Maintainability testing
•Portability testing
•Baseline testing
•Compliance testing
•Documentation testing
•Endurance testing
•Load testing
•Performance testing
•Compatibility testing
•Security testing
•Scalability testing
•Volume testing
•Stress testing
•Recovery testing
•Internationalization testing and
Localization testing
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.
23

24.

GUI TESTING
GUI Testing is a process to test application's
user interface and to make sure that it confirms
the design requirements.
GUI Testing
Text Box
Radio Button
Check Boxes
Command Buttons
Aesthetic Conditions
Validation
Conditions
• Usability Conditions
• Data Integrity
Conditions
• Date Field Checks
24
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

25.

USABILITY TESTING
Usability Testing analyses user
reaction to a product. It checks
whether the product is useful, easy to
use and learn. All in all, it must
satisfy users needs.
25
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

26.

PERFORMANCE TESTING
Performance Testing: The goal of performance testing is not to find bugs, but to
eliminate bottlenecks and establish a baseline for future regression testing.
Load Testing: To verify application behavior under normal and peak load conditions
Stress Testing: To determine application’s behavior when it is pushed beyond normal or
peak load conditions.
a. double the baseline number for concurrent users/HTTP connections
Volume Testing: Testing where the system is subjected to large volumes of data.
a. testing a word processor by editing a very large document
b. testing a printer by sending it a very large job
Capacity Testing: To determine how many users and/or transactions a given system will
support and still meet performance goals.
Soak Testing: involves testing a system with a significant load extended over a significant
period of time, to discover how the system behaves under sustained use.
26
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

27.

PERFORMANCE TESTING
If chair is designed for
100 kg weight, and my
weight is 70 kg then that
testing is called as normal
testing.
If my weight is 100 kg
then that testing is called
as load testing.
If my weight is 120 kg
then that testing called as
stress testing
27
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

28.

INSTALLATION TESTING
Installation / Un-installation: This testing checks correctness of application
installation / un-installation under the environment very close to the real
one.
An Installation Test investigates the effects on the application and on the
target execution system when installed and executed on the target system.
Original installation on a clean target system, update installation on a preexisting installation, and re-installation options are explored.
The Un-installation Test investigates the effects on the target system when
the previously installed application is removed.
28
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

29.

LOCALIZATION TESTING
Globalization Testing (Internationalization Testing): The goal is to detect
potential problems in application design that could inhibit globalization. It
makes sure that the code can handle all international support without
breaking functionality that would cause either data loss or display problems.
Localizability Testing: verifies that you can easily translate the user
interface of the program to any target language without re-engineering or
modifying code.
Localization Testing: checks the quality of a product's localization for a
particular target culture/locale.
29
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

30.

OTHER NON-FUNCTIONAL SOFTWARE TESTING
TYPES
Configuration: Thanks to configuration testing the compatibility between
software and hardware is tested. As a rule, software product is created to
work in at most various environments. These tests are conducted in order to
define a product’s interaction with hardware and software environment.
Products’ reaction to different settings can be checked as well.
Compatibility: Compatibility testing checks functional possibilities and
stability of a product under supported browsers (if talking about webapplications), OS and other third party software that is used by our product.
30
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

31.

RECOVERY TESTING
Recovery Testing is the activity of testing how well an application is able to
recover from crashes, hardware failures and other similar problems.
While an application is running, suddenly restart the computer, and
afterwards check the validness of the application's data integrity.
While an application is receiving data from a network, unplug the
connecting cable. After some time, plug the cable back in and analyze the
application's ability to continue receiving data from the point at which the
network connection disappeared.
Restart the system while a browser has a definite number of sessions.
Afterwards, check that the browser is able to recover all of them.
31
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

32.

SECURITY TESTING
Security Testing is a process to determine that an information system
protects data and maintains functionality as intended.
The six basic security concepts that need to be covered by security testing
are:
Confidentiality
Integrity
Authentication
Availability
Authorization
Non-repudiation
32
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

33.

STRUCTURAL TESTING
- Testing of software structure/architecture
• The structural testing is the testing of the structure of the system or
component.
• Structural testing is often referred to as ‘white box’ or ‘glass box’ or
‘clear-box testing’ because in structural testing we are interested in
what is happening ‘inside the system/application’.
• In structural testing the testers are required to have the knowledge
of the internal implementations of the code. Here the testers
require knowledge of how the software is implemented, how it
works.
33
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

34.

STRUCTURAL TESTING
• During structural testing the tester is concentrating on how the
software does it. For example, a structural technique wants to know
how loops in the software are working. Different test cases may be
derived to exercise the loop once, twice, and many times. This may
be done regardless of the functionality of the software.
• Structural testing can be used at all levels of testing. Developers use
structural testing in component testing and component
integration testing, especially where there is good tool support for
code coverage. Structural testing is also used in system and
acceptance testing, but the structures are different. For example,
the coverage of menu options or major business transactions could
be the structural element in system or acceptance testing.
34
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

35.

CHANGE RELATED TESTING
Regression
It is the selective re-testing to detect faults introduced during
modification of a system or system component, to verify that
modifications or bug-fixing have not caused new defects, and to verify
that a modified system or system component still meets its specified
requirements.
Confirmation testing or retesting
Confirmation testing is the first to be performed after defect
correction. It is done to ensure that the defect has indeed been
successfully removed. The test that originally unveiled the defect by
causing a failure is executed again and this time it should pass without
problems.
35
Types
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

36.

QUALITY AND ITS CRITERIA
Quality is property which shows if a product meets its requirements or
not. And quality is defined on the base of using exactly established
metrics and criteria.
There are two aspects of the quality: product quality and process quality.
Some quality criteria assumptions:
• If customer accepts the implemented product this is enough to say that
the quality of the product is good.
• Product quality is defined by good processes (development,
management, testing etc.)
• If the product meets its requirements – this is good product.
• Good product has some advantages and doesn’t have critical problems.
36
Quality and Its Criteria
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

37.

QUALITY CRITERIA EXAMPLES
Bob Martin on «Agile 2008» had mentioned the following universal metric
for any documentation review: «WTF per minute». You only need a timer
and a counter. Start the timer and read the document. Every time you find
something strange – increment the counter.
37
Quality and Its Criteria
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

38.

QUALITY CRITERIA EXAMPLES
Test coverage not less than 70%
100% of P1 bugs fixed, 90% of P2 bugs fixed, 25% of P3 bugs fixed.
All new functionality and changes to existing one must be implemented
fully according to defined requirements and in time.
Overall passed tests percentage is equal or greater than some value:
X = (Passed/Executed)*100%
38
Quality and Its Criteria
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.

39.

Thank you for your attention!
Questions?
39
Test Team Roles and Responsibilities
® 2016. ITSCHOOL-HILLEL . ALL RIGHTS RESERVED.