Similar presentations:
Risk-based testing
1.
RISK-BASED TESTINGHELEN VOROBEI
OCTOBER, 2017
CONFIDENTIAL
1
2.
INTRODUCTIONHelen Vorobei
Quality Architect,
Testing Competency Center Expert
10 years in testing
E-mail: [email protected]
CONFIDENTIAL
2
3.
AGENDA1
What is risk and risk-based testing?
2
How to identify risks
3
How to apply risks for regression testing
4
How to apply risks for compatibility testing
CONFIDENTIAL
3
4.
TESTING GOALTo find all defects
Find the most critical issues early
Critical issues
Require more time for fixing
May require great changes in architecture
Required a lot of retesting
May block testing
May affect testing and project delivery
milestones
CONFIDENTIAL
4
5.
WHAT CAN HELP? – RISK ANALYSISAnalyze what areas are risky
Plan testing them more and earlier
Reduce testing effort for areas
where there is less risk
CONFIDENTIAL
Complexity
New Feature
User Traffic
Many bugs
Customer
Impact
Integration
Size of Code
Change
Project
Schedule
People
Issues
5
6.
RISK-BASED TESTING: DEFINITIONRisk-based testing is a testing done for the product based on risks. Risk-based
testing uses risk to prioritize and emphasize the appropriate tests/testing types/
test configurations/.. during test execution.
Risk is the probability of occurrence of an undesirable outcome, e.g. issue/bug
Risk attributes:
• Impact - cost when a risk does occur
• Probability - chance that a risk will occur
CONFIDENTIAL
6
7.
HOW TO IDENTIFY RISKSCONFIDENTIAL
7
8.
RISK IMPACTWe get 10 new features for testing. Which bugs will be most important?
System down
Crash
Something that users use frequently
Main business flow
Legal consequences
Everything what many people use and what impacts business – talk to PO/ BA /business
CONFIDENTIAL
8
9.
RISK PROBABILITYWhich features probably will have more bugs than others?
Features with ambiguous requirements
Features with a lot of issues in the past
Technological complex features
Logically complex features
Features that implemented by New /junior developers
Features that were developed in a rush
Features that use new technology for the team
Features with a lot of changes
What else?
CONFIDENTIAL
9
10.
DEFECT ANALYSISThere is not much difference between application areas. Do they have the same risks ?
How to prioritize their testing?
Look at defects: In areas with a lot of defects, probably there are still more
A lot of defects may indicate high risks like:
- poor code quality
- development complexity
- a log of changes were done
- developed in rush
…
=> So areas with a lot of defects may need more testing then others
CONFIDENTIAL
10
11.
HOW TO APPLY RISKSCONFIDENTIAL
11
12.
RISK-BASED APPROACH FOR REGRESSION TESTINGInput: You have 600 test cases and need 20 days to run regression
Problem: Can you complete regression in 10 days?
What to do:
• analyze how many regression defects are found before and their severity
• analyze defect distribution by product components
• analyze where changes were done, evaluate these changes size and complexity
• ask developers what can be affected by new changes
CONFIDENTIAL
12
13.
RISK-BASED APPROACH FOR REGRESSION TESTINGCan we reduce testing effort for these cases?
Definitely, Yes!
Looks No!
Need analyze deeply
CONFIDENTIAL
13
14.
RISK-BASED APPROACH FOR REGRESSION TESTINGAnalyze defects distribution between application components and different testing types
4 the most risky areas with the greatest number of defects
-high probability to find more defects during new testing phase
CONFIDENTIAL
14
15.
RISK-BASED APPROACH FOR REGRESSION TESTINGWhere new changes were done? What is their complexity? What is size
of changes? Do they impact other functionality?
High risk to new
defect appearing in
areas where a lot of
changes were done
and impact high
CONFIDENTIAL
15
16.
RISK-BASED APPROACH FOR REGRESSION TESTINGCombine risk analysis by defects and by new changes and impact
CONFIDENTIAL
16
17.
RISK-BASED APPROACH FOR REGRESSION TESTINGPlan less testing in areas where there is less risk and more testing in areas where there
is high risks
• But not exclude any component from testing at all
What we get in result - reduce regression by 40%!
CONFIDENTIAL
17
18.
RISK-BASED APPROACH FOR TEST CONFIGURATIONS5-10 configurations are in scope. Do we need to repeat the same tests on all
configurations? – Often No!
1. Set up priority based on usage statistic
2. Analyze defects distribution between configurations
3. Distribute tests between configurations and testing types
CONFIDENTIAL
18
19.
RISK-BASED APPROACH FOR TEST CONFIGURATIONS1. Set up priority based on user usage statistic:
Use Analytics to get % of user sessions for top configurations
CONFIDENTIAL
19
20.
RISK-BASED APPROACH FOR TEST CONFIGURATIONS2. Analyze defects distribution between configurations:
Number of Prod defects and total number of defects by different configurations
Calculate %
CONFIDENTIAL
20
21.
RISK-BASED APPROACH FOR TEST CONFIGURATIONS3. Distribute configurations between tests, testing types and testing phases
More and earlier test more risky configurations
Test new features on High priority configurations (for example, divide browsers
between testers)
Regression testing on medium priority and just smoke tests on low priority congs
CONFIDENTIAL
21
22.
WHERE ELSE RISK-BASED APPROACH CAN BE APPLIED?Reorder development plan
Refocus testing activities
start with most complex and risky areas
allocate more time on risky activities, for example, requirement analysis, optimize
efforts on not-risky (check list instead of test cases for not-complex functionality)
Everywhere!
CONFIDENTIAL
22
23.
REAL STORY FROM ONE OF EPAM PROJECTSWHAT WAS BEFORE
• Regression testing took 3 months by team of 10 testers
• Development continued in rush and regression scope was growing
WHAT WAS DONE
• Analyzed regression defects showed that their main reasons are
Not fully tested features during new feature testing
Missed integration between functions
• Team defined more strict DoD to ensure that testing complete
• Change approach for regression testing to scenario-based
WHAT GOT IN RESULT
• Regression testing took 3 weeks execution
• Product quality increased
CONFIDENTIAL
23
24.
HOW TO START WITH DEFECT ANALYSIS?1.
Define what information is important for you: components, configurations, testing types,
testing phase, etc.
2.
Specify bug reporting rules and share within the team, explain why you need this
3.
Perform analysis on regular basis.
4.
Use Excel and pivot tables, chart and diagrams: export defects for some period with all
fields. Period can be defined by releases – last 1-2 releases, for example.
5.
Revise results, ensure that your optimization/ changes in test strategy/ regression testing/
configuration testing do not affect product quality
CONFIDENTIAL
24
25.
MORE DETAILS ABOUT DEFECT ANALYSISExample of bug report ->
- Component tracked in component field
- Regression/UAT/automation/ new feature by labels
It is important to agree the rules what and how
to fill additionally to mandatory fields in bug
reports
CONFIDENTIAL
25
26.
MORE DETAILS ABOUT DEFECT ANALYSISDownload all defects for some period with all
fields into Excel
Example from JIRA
CONFIDENTIAL
26
27.
MORE DETAILS ABOUT DEFECT ANALYSISIn Excel: Insert -> Pivot
Table
In PivotTable Fields:
-
select Priority to
Columns
-
Select Labels to Rows
-
Select Key in Values
MS tutorial link
CONFIDENTIAL
27
28.
HOME TASK1.
Analyze what areas are risky ( or NOT risky) for your application
2.
Propose how testing can be optimized or improved based on result analysis
CONFIDENTIAL
28