Workshop on “Advances & Innovations in Software Testing” Importance of Test Automation in System Testing

1. Workshop on “Advances & Innovations in Software Testing” Importance of Test Automation in System Testing

Workshop on
“Advances & Innovations in Software
Testing”
Importance of Test Automation in
System Testing
Date: Tue, May 8th ‘07
Venue: FedEx Institute of Technology, Memphis

2. Test Automation, Tools and Enablers

Speaker:
Sudheer Mamadapur, PMP

3. Agenda

Introduction
Test Automation
Tools & Enablers
Challenges
About RelQ
RelQ’s Experience
Conclusion
Q&A
RelQ Software © 2007
3

4. Introduction

5. About the Speaker

Sudheer Mamadapur
Project Management Professional
Test Manager – RelQ Software Inc.
8 & half years of experience in RelQ focused on Software V&V and
Test Automation.
Architected framework for Test Automation across various
domains, technologies, automation methodologies and tools.
Roles in RelQ have varied from taking Test Automation challenges
head-on to performing duties as Test Lead, Onsite/Offshore
Coordinator, Test Automation Practice Manager and Test Manager.
RelQ Software © 2007
5

6. System Testing

System testing is testing conducted on a complete, integrated system
Types of System Testing:
Functional
Usability
Performance (Load, Stress, Volume)
Compatibility
Security Testing
Smoke Testing
Exploratory/Adhoc Testing
Regression Testing
Reliability Testing
Recovery Testing
Installation
Accessibility
RelQ Software © 2007
6

7. Why Test Automation?

Software Testing is an integral, costly and time-consuming
activity in the SDLC.
The challenge of the day for IT companies is to improve
productivity and reduce the cost of Testing.
Test Automation plays a pivotal role in effective testing and
reduction in test effort.
Changes in environment like new patches, service packs etc.
also call for testing/test automation.
Keep pace with development activities
A strategic approach for Test Automation using tools, framework
and methodologies will improve test coverage in regression
cycles and reduce the test effort in subsequent release cycles.
“Rapidly changing software is a moving target, but Test Automation
can still hit it“
RelQ Software © 2007
7

8. Test Automation

9. Benefits of Test Automation

Cost Effective and Efficient Process
Consistent and Accurate Test Cycles
Reduces Overall Test Cycle Time
Mitigates Risks of Manual Testing
Increases Overall Product Quality
Early Time to Market
RelQ Software © 2007
9

10. What to Automate?

Governs the
Quality of tests
Analyze
Design
Intellectual
Performed
once
Good to
Automate
Construct
Test
Deploy /
Maintain
RelQ Software © 2007
Repeated many
times
Clerical
10

11. What to Automate? (contd…)

Which & What Test Should be Automated First ?
Most important tests
A set of breadth tests
Test for the most important functions
Tests that are easy to automate
Tests that will give the quickest payback
Tests that are run more often
Alternatively, test candidates that are identified as
part of risk-based testing strategy may as well be
considered for test automation.
RelQ Software © 2007
11

12. Test Automation Lifecycle

Tools
Processes
6. Maintain
5. Deploy
4. Test
Skilled Resources
RelQ Software © 2007
1. Analyze
2. Design
3. Construct
Methodologies
12

13. RelQ’s Approach

System study
and confirm
technical requirement
Tool evaluation
and selection
Application
navigation
using the tool
Application
updates
Application support
/development
Review results
and report
the defects
Selection of test
automation
methodology
Review and
approve
scripts
Perform tests/
regression testing
Build test scripts
and test drivers
Enhancement of
test scripts
Test summary report
© RelQ Software
RelQ Software © 2007
13

14. Tools & Enablers

Tools & Enablers

15. Tools

Mercury
Quick Test Professional
WinRunner
IBM Rational
Rational Robot
Functional Tester
Borland
Silk Test
Compuware
QA Run
AutomatedQA
TestComplete
RelQ Software © 2007
15

16. Tool Characteristics & Criteria

Tool Characteristics & Criteria
Capability
Capacity
Compatibility
Learnability
Operability
RelQ Software © 2007
16

17. Enablers

Methodologies
Data Driven
Functional Decomposition
Keyword Driven
Framework
Test Framework
Process Framework
RelQ Software © 2007
17

18. Level 1: Basic Automation

19. Record & Playback

Record & Playback
Approach:
Test scripts are recorded on the initial version of the
application.
The same scripts are executed on the next version.
The scripts need some modification for the changes
happening on the application during every version of the
application.
The test scripts repository keeps growing as the application
goes through changes.
RelQ Software © 2007
19

20. Record & Playback (contd…)

Record & Playback (contd…)
Application
Version 1
Application
Version 2
Playback
Application
Version ‘n’
Modify
Playback
Record
Test Scripts
An automation tool generates
scripts by recording user actions.
The generated scripts can be
played back to reproduce the exact
user actions.
Advantages:
• Less effort for automation
• Quick returns
• Does not require expertise on tools
Limitations:
• High dependency on the GUI of
AUT
• Difficult to maintain the scripts
RelQ Software © 2007
20

21. Data-Driven Methodology

Approach:
Scripts are freed from hardcoded data.
Test input and output values are read from data files (data
pools, ODBC sources, cvs files, Excel files, DAO objects,
ADO objects, and such) and are loaded into variables in
captured or manually coded scripts.
Variables are used for both input values and output
verification values.
Navigation through the program, reading of the data files,
and logging of test status and information are all coded in
the test script.
RelQ Software © 2007
21

22. Data-Driven Methodology (contd…)

APPLICATION UNDER TEST
Test Mgmt. tool
Test Case
Mgmt tool
Automation
tool
Driver
Scripts
Defect
Mgmt tool
Config
Files
Scripts
Data
Files
Object
Repository
RelQ Software © 2007
Libraries
Test
Results
Advantages:
• Reduces the overall number of scripts
needed to cover the test cases
• Data is separated from script and
stored in a data file; easy to maintain
• The volume of test data and its
combination can be increased for
exhaustive testing
Limitations:
• Application must be available to start
scripting
• Highly depends on the GUI of AUT
• When the GUI design changes, the
Automation scripts require change
• Maintenance of the test script and
data files are very high
22

23. Level 2: Advanced Methodologies

24. Functional Decomposition

Create small, independent scripts that represent modules,
sections, and functions of the application-under-test.
Scripts are taken and combined in a hierarchical fashion to
construct larger tests.
Use of this methodology yields a higher degree of
modularization and add to the overall maintainability of the
test scripts.
Advantages:
• Provides separation between data and scripts
• Delivers script reusability
• Provides one maintenance point for each functionality or screen
Limitations:
• Longer implementation time
• Implementation team must have programming background
RelQ Software © 2007
24

25. Keyword-Driven Methodology

Approach:
The discrete functional business events that make up any
application can be described using a short text description
(Keyword) and associated parameter value pairs
(arguments).
By designing Keywords to describe discrete functional
business events, the testers begin to build up a common
library of Keywords that can be used to create Keyword
test cases.
This is really a process of creating a language (Keywords) to
describe a sequence of events within the application (Test
Case).
RelQ Software © 2007
25

26. Keyword-Driven Methodology (contd…)

APPLICATION UNDER TEST
Test Mgmt. tool
Test Case
Mgmt tool
Utility
Libraries
Automation
tool
Defect
Mgmt tool
Driver
Scripts
Config
Files
Action Files
Keyword
Tables
Functional
Libraries
Object
Repository
RelQ Software © 2007
Test
Results
Advantages:
Development of Keywords, Action Files can
begin before the application is ready
The User can easily create the test scenarios
by grouping a set of Keywords in the Action
file.
The User has multiple options of test case
selection for execution.
Minimum coding required for addition /
modification of test cases.
Maintenance is easy, changes in functionality
require changes in Action Files only
Keywords can be reused for other
applications’ testing
Limitations:
Requires extensive programming background
Requires longer implementation time
26

27. Benefits of Automation Methodology

Effort
Initial Development
Ongoing Maintenance
Record/
Replay
DataDriven
Modula
r
KeywordDriven
Methodology
RelQ Software © 2007
27

28. Level 3: Framework

29. Framework

Framework-driven automation is an innovative approach to
software testing that leverages proprietary automation
frameworks and tool sets from leading vendors, supported by
additional programming to maximize the benefits of test
automation.
Types:
Test Framework
Focus on creating reusable test automation components.
Process Framework
Focuses on automation of process in addition to the type of
test.
Hybrid Framework
Most effective framework that combines the features of
process and test frameworks.
RelQ Software © 2007
29

30. Test Framework

DESIGN CONSIDERATIONS
Modularity
The framework to be modularized in line with the
application architecture.
Maintainability
The Scripts to be normalized and freed from any kind
of data within.
Coding Standards to be defined, followed and audited.
Functional decomposition method to be used to
identify commonly used functions and the same to be
kept outside the main script.
The configuration files and data files to be maintained
in user-friendly format.
RelQ Software © 2007
30

31. Test Framework (contd…)

Flexibility
The framework to accommodate automation on
different interfaces; GUI (Web/CS), Character Based
Interface, Mainframe,..
Scalability
Accommodate addition of new test cases to the
framework with minimum effort
Extensibility
The framework to be extended to accommodate
different types of Validations; GUI, Database, File,…
Reusability
Functions / scripts to be designed to facilitate
reusability across tools & technologies
RelQ Software © 2007
31

32. Process Framework

End-to-End Automation
Identification of Test Environment
Installation of Build(s)
Test Set(s) Selection
Test Execution
Result Reporting
Advantages:
Useful when different tests have the similar test setup
steps.
Provide common Interface to Plug-in Multiple types of
tests.
Consistent set of commands to execute different tests.
RelQ Software © 2007
32

33. Test Automation Challenges

34. Challenges

Organization
Poor testing practice
Business:
Unrealistic expectations
Achieving ROI
False sense of security
Application:
Frequent changes to the User Interface / Business Rules
Tool:
Object Recognition Limitations
Support for new technologies (.NET,...)
Technical support from tool vendor / user forum
RelQ Software © 2007
34

35. About RelQ

36. About RelQ

RelQ Software is an Independent Software Verification &
Validation (IV&V) Services Company. RelQ provides
software testing and related Quality Assurance services to
IT Organizations globally, to help them deliver reliable and
quality software on time to their customers.
Established in 1998, RelQ has been a pioneer in this
segment in the Indian IT Industry.
RelQ has around 1,800 person-years of experience in
Software Testing across 350+ projects.
Currently, 800+ RelQ engineers are working on testing
assignments for clients across the globe.
RelQ Software © 2007
36

37. RelQ and Test Automation

RelQ is focusing on Test Automation since 1998.
RelQ has experience of more than 1,200 person-months
on Test Automation across domains, technologies and tools.
The knowledge acquired in the projects has helped RelQ in
defining the methodologies for test automation and improve
upon the same over a period of time.
The metrics collected in the projects has helped RelQ build
reliable models for effort estimation in Test Automation.
RelQ’s experience – 75% of the application testing projects
required Test Automation!
RelQ Software © 2007
37

38. RelQ’s Experience in Test Automation

39. Levels of Test Automation

Level 3
ADVANCED
METHODOLOGIES:
Data-Driven,
Keyword-Driven,
Hybrid
Level 2
Level 1
RelQ Software © 2007
FRAMEWORK:
Test / Process / Hybrid:
Methodologies, Libraries,
Re-usable components
BASIC:
Record & Playback,
Functional
Decomposition
39

40. Statistics

Level
Type of Applications
Effort spent on
Automation
Tools Used
Methodology
Level 1
Client Server,
Web
200 personmonths
Rational Robot,
WinRunner,
SilkTest,
QTP
Record &
Playback,
Functional
Decomposition
Level 2
Client Server, Web,
Mainframe, Devices,
ERP Packages
700 personmonths
Rational Robot,
WinRunner,
SilkTest,
QTP, Test
Complete
Data Driven,
Keyword Driven,
Hybrid
Level 3
Multilayered, Complex
Applications
300 personmonths
QC, BPT, QTP,
SilkTest, Perl,
Shell Scripting
Framework,
Libraries,
Reusable
Components
RelQ Software © 2007
40

41. Conclusion

System Testing is a costly & time-consuming activity
Test Automation helps overcome the difficulties of
manual testing thus reducing the effort and time
required for testing.
A framework based approach for test automation
supports maintainability, extensibility, scalability, …
A strategically developed framework based automation
suite can even be used for other similar
applications/platforms thus maximizing the ROI.
Integrated with a test management solution, provides
complete solution for automated testing, reporting, test
coverage, defect logging, …
RelQ Software © 2007
41

42. Q & A

Q&A

43.

End of Presentation