Testing in the Digital Age: DevOps, ChatOps and Automation to support Testing

1. Testing in the Digital Age: DevOps, ChatOps and Automation to support Testing

DevOps, ChatOps and Automation to support Testing
Paul Gerrard
[email protected]
gerrardconsulting.com
@paul_gerrard
Testing in the Digital Age:

2.

Helping clients transform their testing through
INNOVATION, COACHING and LEADERSHIP
Our CLIENTS
– Want to be agile rather than follow Agile dogma
– Have a pragmatic approach and are focused on delivery
– Want a solution that fits, not a badly fitting suit.
Intelligent Definition and Assurance
Slide 2

3. Agenda

Digital Transformation
Digital is a Business Initiative
New Model Testing
Shift-Left
Collaborative Requirements
Continuous Delivery and DevOps
The Tools Landscape
ChatOps
Test Automation – Ambition and Reality
Digital Testing Strategy
Surviving Continuous Delivery and DevOps.
Intelligent Definition and Assurance
Slide 3

4. Introductions

• Say hello to your team
• Give your team a name (nameyourapp)
• Please try to login to our Slack site:
–
–
–
–
https://chatopsdemo.slack .com
Username: [email protected]
Password: p4ssw0rd
My card has the number xx to use
• What do you want to talk about?
Intelligent Definition and Assurance
Slide 4

5. This class is about thinking for yourself

• Pressure on testers – greater than ever
• The IT industry is responding to Digital – a
business initiative with new approaches
– Shift-left, DevOps, continuous, pervasive
automation, experimentation, analytics
• These approaches are not standardised, they
are changing and new tools appear every week
• I can’t give you a ‘formula’ – but I can give you
some ideas and thinking tools.
Intelligent Definition and Assurance
Slide 5

6. Digital Transformation

7. Revolution!

• Lots of hype around “Digital” bombarding
business and IT
• Digital Transformation programmes are
affecting business across all industry and
government sectors
• There is no doubt that it also affects people in
their daily lives.
Intelligent Definition and Assurance
7

8. Scope

• Digital includes traditional IT but includes:
–
–
–
–
–
–
–
Mobile anything
The Internet of Things
Autonomous vehicles
Our home, workplace, public and private spaces
Robots (physical)
Bots (software)
Artificial Intelligence, Machine Learning, Deep Learning
(not limited to corporates)
• All our lives will be affected.
Intelligent Definition and Assurance
8

9. Digital Transformation

• Internet users:
– 2005: 1bn, 2010: 2bn,
2014: 3bn
• Affordable technological advances:
–
–
–
–
Small devices – huge range; local/power networking
AI finally delivers in the form of Machine Learning
Virtual and Augmented Reality-based systems
Robotics, drone technology and 3D printing
• Almost all businesses have committed to these
technologies…
• … and they are calling it Digital Transformation.
Intelligent Definition and Assurance
9

10. Unprecedented project ambition

A Smart City is…
• “… an urban development vision to integrate
multiple information and communication
technology (ICT) and IoT solutions in a secure
fashion to manage a city’s assets – the city’s
assets include, but are not limited to, local
departments' information systems, schools,
libraries, transportation systems, hospitals, power
plants, water supply networks, waste
management, law enforcement, and other
community services.” (Wikipedia)
Intelligent Definition and Assurance
12

11. The most complex systems ever?

Smart Cities: the largest systems
ever built?
• Nodes and endpoints: from a million to billions
• Bigger, more complex than anything before
• Connected citizens and many of the systems:
– Move in the realm of the city and beyond
– Interact in unpredictable ways
– Citizens are not hand-picked like the military;
crooks, spies and terrorists can usually come and
go as they please.
Intelligent Definition and Assurance
13

12. A Smart City is…

Systems of Every Scale
• Scale of Digital ranges from the trivial to the
largest systems ever attempted
– Home automation product (10-30 components)
– Factory automation, monitoring and management
(several thousand components)
– Smart City (probably millions)
• “A Digital Ecosystem is a distributed, adaptive,
open socio-technical system with properties
of self-organisation, scalability and sustainability
inspired from natural ecosystems”.
Intelligent Definition and Assurance
15

13. Smart Cities: the largest systems ever built?

Systems with Social Impact
• Digital systems will have a social impact on all
citizens who encounter them
• Huge consequences as systems become more
integrated with the fabric of society
• Systems already monitor our every move, our
buying, browsing and social activities
• Bots push suggestions of what to buy, where
to shop, who to meet, when to pay bills to us
minute by minute.
Intelligent Definition and Assurance
16

14. Systems of Systems and Ecosystems

Ecosystems of Ecosystems
• The span of Digital covers commerce,
agriculture, health, government, the media in
its all forms, the military …
• Digital affects all industries
• A systems view does not do it justice
• It might be more appropriate to treat Digital
systems as…
– “Ecosystems within Ecosystems”.
Intelligent Definition and Assurance
18

15. Systems of Every Scale

Digital is a Business
Initiative
Agile (an IT initiative) has taken 15
years to get half way
Digital?

16. Systems with Social Impact

Digital and Marketing
• Digital is the buzz-phrase of the moment
– Social, Mobile, Analytics, Cloud (SMAC)
– Consumerization of IT
– Buzzword bingo*
• Agile laggards (gov, large companies) playing
catch-up?
• Marketers in charge: Chief Digital Officers etc.
• S/W development at the pace of marketing.
* https://zoomph.com/blog/top-20-digital-marketing-buzzwords/
Intelligent Definition and Assurance
Slide 20

17. Law enforcement

Frequent/regular s/w delivery is
critical
• Mobile users expect apps to change almost
daily
– New features, offers, opportunities all the time
• Users use the best apps to get what they want
– Not necessarily the best supplier
• Businesses are in an APPS RACE
• Speed of delivery is...
– Partly about pro-action
– Partly about survival.
Intelligent Definition and Assurance
Slide 21

18. Ecosystems of Ecosystems

Automation (not just test) is
critical
• Business wants IT responsiveness (true agility)
– Not necessarily 100s of releases/day
– Rapid, regular turnaround from ideas to software
delivery
• With continuous integration/deployment,
DevOps, developers can now promise
Continuous Delivery
• Testers need to provide Continuous Assurance
• "Automation through the (shortened) lifecycle"
• But effective test automation success is
still hard to achieve.
Intelligent Definition and Assurance
Slide 22

19. Digital is a Business Initiative

"Automation is the future!"
• Heard that before?
• What exactly is possible and impossible with
automation, right here, right now?
• Are Continuous Delivery and DevOps the route to
success?
• Could testing be the bottleneck that prevents success?
• How do testers work in high-paced, automationdominated environments?
• In the rest of this tutorial, I want to explore these and
other questions.
Intelligent Definition and Assurance
Slide 23

20. Digital and Marketing

The old ways won't
work in the future
We need a New Model of Testing
(free from logistics)

21. Frequent/regular s/w delivery is critical

Forget Logistics
(for the time being)
Document or not?
Automated or manual?
Agile v waterfall?
This business or that business?
This technology v that technology?

22. Automation (not just test) is critical

ALL Testing is
Exploratory
We explore sources of knowledge ...
... to build test models ...
... that inform our testing.

23. "Automation is the future!"

How many models can YOU name?
Paul Gerrard
[email protected]
gerrardconsulting.com
Intelligent Definition and Assurance
Twitter: @paul_gerrard
Models Pub Quiz
Slide 27

24. The old ways won't work in the future

Models are innate, essential, human
Intelligent Definition and Assurance
Slide 28

25. Forget Logistics (for the time being)

Judgement, exploring and testing
We explore sources of knowledge to build test models that inform our testing
Our model(s) are adequate
Creates test
models
Exploring
(sources)
Uses test
models
Judgement
Testing
(the system)
We don't yet know what the
We believe that we know what
Our
system should
do.model(s) are not adequate
the system should do.
We can't test yet
We can (automate) test.
(learning what it should do)
(learning what it does do)

26. ALL Testing is Exploratory

Judgement, exploring and testing
We explore sources of knowledge to build test models that inform our testing
Our model(s) are adequate
Creates test
models
Exploring
(sources)
Uses test
models
Judgement
Testing
(the system)
Our model(s)
are not
BTW – Do Developers
explore
theadequate
same way? I think so.
Intelligent Definition and Assurance
Slide 30

27. Models Pub Quiz How many models can YOU name?

Exploration process
Sources
System
under test
Enquiring
Modelling
People
(& you)
Test
Models
Exploration
Definitions
specs/stories
Requirements
Challenging
Sources:
Predicting
Test Models:
People, documents,
experience, system under test
Intelligent Definition and Assurance
Can be documented
or mental models
Slide 31

28. Models are innate, essential, human

Testing process
Reporting
More exploring
Informing
Test
Models
Applying
System
Under Test
Testing
Refining
Interpreting
Revise the
System
Intelligent Definition and Assurance
Slide 32

29. Judgement, exploring and testing

New Model Testing
29 page paper: http://dev.sp.qa/download/newModel
Intelligent Definition and Assurance
Slide 33

30. Judgement, exploring and testing

Some Consequences
There are others

31. Exploration process

Relation to
TDD and BDD?
TDD is not really testing
BDD is modelling using stories

32. Testing process

Test automation from a
different perspective
Automation efforts fail too often
Automation uses different test models

33. New Model Testing

Capabilities
Enquiring, Modelling, Predicting, Challenging
Informing, Applying, Interpreting, Refining
Reporting and Logging

34. Some Consequences

A very different skillset
• Analysis, enquiry and elicitation
• Modelling
• Creation of custom models, using
heuristics, guesses, brainstorming,
ideation, creative thinking
• Custom test design techniques
• Comparison of models, value, advantages,
disadvantages, compromises
• Identification, validation and use of
oracles
• Predicate logic and proof
• Hypothesis and inference
• Socratic method
• Rapid Review and Inspection techniques
• Test case design
• Test models and the meaning of coverage
• Testing as controlled experiment
• Observation, Note taking, recording
Basic data analysis and statistics
Decision-making with incomplete data
Computer forensics
Fault tree analysis
Failure diagnosis
Bug advocacy, triage processes and
negotiation
Meaningful software and test metrics
Visual presentation of data
Reporting and presentation skills
Understanding stakeholders
Test analytics
Risk management, risk-based testing and
decision-making
Critical Thinking
Interpersonal skills
Dealing with uncertainty/fallibility
Intelligent Definition and Assurance
Slide 38

35. Relation to TDD and BDD?

Testing Career Development
(speculative)
Strategic
Test Strategy
Project
Intelligence
Test Assurance
ISTQB, TMMi etc...
Management
Stakeholder
management
Supplier
Management
Analytics &
visualisation
Managing
uncertainty
Test Process
Management
Test Automation
Technical (Excel,
SQL, OS utils etc)
Methodology
Forensics
Interpretation
Critical Thinking
Technical
Scripting/
Programming
Foundations
Exploration
Intelligent Definition and Assurance
Slide 39

36. Test automation from a different perspective

Shift-Left

37. Capabilities

Shift-left
• Teams redistribute responsibility for testing
and collaborate more
• Shift-Left can mean:
– Developers take ownership for their testing
– Testers get involved earlier, challenge
requirements, share examples with users and devs
– No test team and no testers
• There is no 'one true way' of course.
Intelligent Definition and Assurance
Slide 41

38. A very different skillset

Shift-Left is not new
Paul Herzlich's WModel 1993
• Shift-Left is mostly about bringing the thinking about testing earlier in the process
• So, all we do is get involved earlier and ask awkward questions?
• Is it really as simple as that? Well, not quite.
Intelligent Definition and Assurance
Slide 42

39.

Shift-Left is Not New
• “Test early, test often”: a mantra for 25 years
• The underlying principle:
– Sources of Knowledge should be challenged or tested
– In a staged project, this might involve formal reviews
– In an Agile project, ‘three Amigos’ tester, developer
and user/BA suggest scenarios (or examples) that
challenge requirements before code is written
• Shift-Left is really “thinking about testing earlier”.
Intelligent Definition and Assurance
Slide 43

40. Shift-Left

– it’s all about feedback
• Testers provide feedback – whenever possible
– Get involved early – as early as you can
– Challenge through example
• Software development is knowledge acquisition
– Knowledge is gathered throughout the project and
evolves over time
– The goal is to assure this knowledge and to ensure it
is trusted before it is frozen in code
• Shift-Left is not a threat; it is an opportunity to
make a bigger, better contribution.
Intelligent Definition and Assurance
Slide 45

41. Shift-left

Discussion: How often do apps
change?
• How often are these apps
released?
–
–
–
–
–
–
–
–
–
–
Facebook
LinkedIn
eBay
Amazon
NatWest bank
Lloyds Bank
Nationwide Bank
etsy
Google search
your app
• Release frequency
–
–
–
–
–
–
–
–
–
–
2 weeks (mostly)
7-14 days
Monthly
2-4 weeks
4-6 weeks
1-3 months
2-8 weeks
2 weeks
4-6 weeks
• Stats from appannie.com
Intelligent Definition and Assurance
Slide 46

42. Shift-Left is not new

Collaborative
Requirements

43. Shift-Left is Not New

Collaborative requirements
• Testers can make a huge contribution to the
requirements phase
• In most contexts, user or business stories are
used to capture them
• For the purpose of the next few slides:
– 'Requirements' could mean documented or
undocumented sources of knowledge
– A 'story' summarises a system feature and illustrates it
use with scenarios.
• I‘ll use the Gherkin (Cucumber) format to
illustrate its use (heard of Cucumber?)
Intelligent Definition and Assurance
Slide 48

44. What is Driving Shift-Left?

Using Stories to
Articulate/Validate
Requirements
Chapter 4 of the Business Story
Pocketbook
Download here:
https://tkbase.com/resources/viewResource/19

45. Shift-Left – it’s all about feedback

Intelligent Definition and Assurance
Slide 50

46. Discussion: How often do apps change?

Structured stories (other variations exist)
Feature:
As a
I want
So that
ship orders
Story Header
orders clerk
to acknowledge and ship the order
we fulfil a book order
Scenario:
Given
And
When
Then
And
ship a single book from stock
I select a valid order
the ordered book is in stock
I choose ‘acknowledge and ship’
order status is changed to ‘shipped’
an address label is printed
Key word
Story text
Each Story has multiple Scenarios
Scenarios can be data driven
Intelligent Definition and Assurance
Slide 51

47. Collaborative Requirements

Anatomy of a business story
header
Feature
A label for the FACILITY a user needs to support
their business objective
As a…
The ROLE of the user who needs to achieve the
goal
I want (to)… The TASK the user wants to perform with the help
of the system
So that…
The GOAL of the user
The story brings together these aspects so we can view the feature from
different viewpoint to explore it
Note that roles can sometimes vary, but it is often better to reference ‘personas’ that have multiple roles.
Personas could be “18 year old male gamer” or “65 year old female retired nursery school teacher” for
example.
Intelligent Definition and Assurance
Slide 52

48. Collaborative requirements

Anatomy of a scenario
Given… Pre-conditions that define the state of the system or
scenario when the user performs the task
When… The values, inputs, commands or actions executed by
the user to complete the task
Then…
The outcome of the task (given the preconditions and
the user inputs, actions etc.)
• The parallel with test cases is obvious:
• given=precondition(s), when=steps, then=outcome(s)
• A scenario maps directly to a test case – but we haven’t used the
word test yet.
• If I do – stop me.
Intelligent Definition and Assurance
Slide 53

49. Using Stories to Articulate/Validate Requirements

Stories may have many scenarios
Feature: Ship an Order
In order to fulfil a book order
As a orders clerk
I want to acknowledge and ship the order
Scenario: ship a single book from stock
Given I select a valid order
And the ordered book is in stock
When I choose ‘acknowledge and ship’
Then order status is changed to ‘shipped’
And an address label is printed
Scenario: advise a book is out of stock
Given I select a valid order
And the ordered book is out of stock
When I choose ‘message the purchaser’
Then Enter message to purchaser advising the order status
And an email is sent to the purchasers email address
Scenario: advise an item is discontinued
Given I select a valid order
And the ordered book is discontinued
……
Etc. etc.
Intelligent Definition and Assurance
Slide 54

50.

New Model Testing
We use examples,
scenarios to challenge
requirements here
Scenarios are examples
of behaviour
Intelligent Definition and Assurance
Slide 55

51. Structured stories (other variations exist)

Exercise: what needs definition?
Jot down the terms to define
1. A customer order will have a unique order
reference a customer identifier, an order
date and a required-by date.
2. Each order generates multiple order items
each of which has a unique ID, a product
identifier, an order quantity, a unit
price, a unit weight, a promised delivery
date and required-by date.
3. The total order price, earliest and
latest delivery date, total weight will
be calculated from the items on the
order.
Do you see any ambiguities?
Intelligent Definition and Assurance
Slide 56

52. Anatomy of a business story header

Exercise: what needs definition?
1. A customer order will have a unique order
reference a customer identifier, an order
date and a required-by date.
2. Each order generates multiple order items
each of which has a unique ID, a product
identifier, an order quantity, a unit
price, a unit weight, a promised delivery
date and required-by date.
3. The total order price, earliest and
latest delivery date, total weight will
be calculated from the items on the
order.
Intelligent Definition and Assurance
Slide 57

53. Anatomy of a scenario

What other questions might you
ask?
1.
2.
3.
'customer order will have…' means what?
unique order reference’ unique in what context?
Order date’: date placed, or date recorded or other date to be
defined?
4. Each order generates’ – means what?
5. Can an order have zero items? Is there a limit to how many items?
6. Can order quantity be negative, non-zero, is it a decimal or
integer?
7. Must dates be weekdays or can they be weekends? Are there any date
rules to apply?
8. What currency are prices in?
9. What units is weight measured in?
10. Is deliver-date the last required-by date or is it calculated some
other way?
11. Other definitions? ‘Customer’, ‘required-by’, ‘weight’, ‘unit’,
`promised’, ‘total (price)’, ‘product’
12. Where do these pieces of data come from? Where are they stored?
Where do they ‘go to?
Intelligent Definition and Assurance
Slide 58

54. Stories may have many scenarios

Typical questions to ask of a
requirement
What do the nouns and verbs actually mean?
What features are being described here?
What outcomes do these features provide?
What scenarios (normal, extreme, edge and
exceptional) should they cope with?
• Are all outcomes predictable from the text?
• Are all outcomes unambiguous?
• Is anything (definitions, features, scenarios or
outcomes) missing?
Intelligent Definition and Assurance
Definition
Feature
Outcome
Scenarios
Prediction
Ambiguity
Missing
Slide 59

55. New Model Testing

D e F O S PA M
Definitions
Features
Outcomes
Scenarios
Prediction
• Ambiguity
• Missing
Identify the terms and define
One story per feature
One scenario per outcome
One scenario per scenario
Can’t predict? Scenario +
guess outcome
Two stories with conflicts
Add a story as a suggestion
See “Business Story Method”, page 41
Intelligent Definition and Assurance
Slide 60

56. Exercise: what needs definition? Jot down the terms to define

Exercise: D e F O S P A M
Use DeFOSPAM
to find problems
with this
requirement
• Create some
examples or
scenarios to
illustrate
problems.
• The calculator will accept three inputs: a
number A, an operator O and another
number, B.
• The calculator will validate the numbers A
and B as valid in the range -1000,000,000 to
+1000,000,000
• The operator may be one of the following:
"+" (plus), "-" (minus), "*" (multiply) or "/"
(divide)
• The calculator will perform the calculation
according to standard arithmetical rules
• The calculator will print the result as a real
number with up to 20 significant digits.
Intelligent Definition and Assurance
Slide 61

57. Exercise: what needs definition?

Did you find any
anomalies?
Do you want to share them?

58. What other questions might you ask?

Continuous Delivery

59. Typical questions to ask of a requirement

From traditional delivery…
… to Continuous Delivery
Intelligent Definition and Assurance
Slide 64

60. D e F O S P A M

The Deployment Pipeline
• Automated Unit
tests
• Automated
Acceptance Tests
• Manual User Tests?
Intelligent Definition and Assurance
Slide 65

61. Exercise: D e F O S P A M

Introducing Continuous Delivery
• Rigorously scoped features
– If a feature can't be squeezed into the next
release, split it into smaller features
• Validated requirements – thru example
– BDD/Three Amigos/Collaborative requirements
• Gherkin given/when/then scenarios
– Validate the requirements
– But can also become automated tests.
Intelligent Definition and Assurance
Slide 66

62. Did you find any anomalies?

Introducing Continuous Delivery 2
• Test-Driven Development – all tests pass
• Commits must work – broken builds or tests
in CI are the teams top priority
• Everything under change control
• (Almost) Everything automated
• Rapid/regular feedback requires rapid/regular
reaction.
Intelligent Definition and Assurance
Slide 67

63. Continuous Delivery

in pictures
http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/

64. From traditional delivery…

If it Hurts, Do it More (Often)
• Tasks (like testing) provide feedback; regular rapid feedback means
that problems can fixed quickly
• If we do a hard activity more frequently, We get better at it, maybe
enough to automate it.
Intelligent Definition and Assurance
Slide 69

65. The Deployment Pipeline

OK, we’re going to implement CD
• Boss: “We’re want to automate EVERYTHING!”
• Break down the test activities you are involved in
day to day
– Which ones can be automated?
– And which ones can’t?
– Discuss in your group, make a list of each.
Intelligent Definition and Assurance
Slide 70

66. Introducing Continuous Delivery

Exercise: what test activities can be
automated and what cannot?
What can be automated?
What can’t be automated?
Intelligent Definition and Assurance
Slide 71

67. Introducing Continuous Delivery 2

It’s not about
automation,
it’s about trust
A Deployment Pipeline depends on a
Trusted Requirements Pipeline

68. Continuous Delivery in pictures http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/

If a requirement drives today’s
delivery it must be trusted
• Continuous delivery is a hungry beast that
eats requirements
• Requirements must be trusted to be mature,
complete and coherent enough to deliver the
business value envisaged by stakeholders
TRUSTED, NOT PERFECT
Intelligent Definition and Assurance
Slide 73

69. If it Hurts, Do it More (Often)

Tests, Automation and Trust
• There is debate around:
– The meaning of checking and testing
– The reliance we can place on testers, on checks
and automation
• In most environments we can't place all our
faith in automated checks
• We need to look more closely at our testing.
Intelligent Definition and Assurance
Slide 74

70. OK, we’re going to implement CD

Four types of checks?
1. Checks that can be automated by developers as
part of their component-level check-in and
Continuous Integration processes
2. Checks that can be automated (typically by
system testers) to exercise API-level, link or
end-to-end transactions
3. Tests that can perform compatibility checks to
demonstrate compatibility across browsers,
operating systems, platforms
4. Tests that can only be performed by a human.
Intelligent Definition and Assurance
Slide 75

71. Exercise: what test activities can be automated and what cannot?

Typical questions and objections
• When do we create the test automation?
• There are too many system tests, how can we
automate them all?
• When we find bugs, there isn’t time to rerun
our systems tests
• …
Intelligent Definition and Assurance
Slide 76

72. It’s not about automation, it’s about trust

Shifting tests left
Move manual user and
system tests earlier and
automate
• User/acceptance tests
• GUI/System Tests
• API/Service tests
• Unit Tests
Intelligent Definition and Assurance
Slide 77

73. If a requirement drives today’s delivery it must be trusted

New feature and regression testing
Test new code
Regression test
existing code
Automated tests
Tested once as new
code, and many times
regression tested
Intelligent Definition and Assurance
Slide 78

74. Tests, Automation and Trust

Development Patterns

75. Four types of checks?

Three development patterns
Structured
Agile
Continuous
Autonomous
Intelligent Definition and Assurance
Slide 80

76. Typical questions and objections

Characteristic
Structure
Pace/cadence
Leadership
Definition
Testing
Automation
Measurement
Governance
Summary
What is the organisational structure of the
project team?
What drives the rate of decision making?
Who do decisions depend on?
How is the team managed/directed? What
style of leadership is involved?
How is requirements knowledge captured? In
what format?
How is testing (mostly) performed? Scripted,
exploratory, automated?
When is automation used? Who leads the
automation effort?
What/how is project measurement
performed?
What
form
doesandgovernance
take?
Intelligent
Definition
Assurance
Slide 81

77. Shifting tests left

Profiles of the three patterns
Characteristic
Structure
Pace/cadence
Agile
Continuous
Autonomous Production Cell
Team decision Feedback
Definition
Structured
Managed team
Business
decision
Project
Managed
Fixed spec
Testing
Automation
Measurement
Governance
Scripted
Retrospective
Pervasive
Bureaucratic
Exploratory
Developer led
Avoided
Trust-based
Leadership
Guided
Research
Dynamic spec
Intelligent Definition and Assurance
Line Managed
Executable
Specs
Automated
Pervasive
Analytics
Insight-Driven
Slide 82

78. New feature and regression testing

Not three patterns;
There are many
You have to work out your own
hybrid approach that suits your
organisation

79. Development Patterns

DevOps
an overview

80. Three development patterns

Exercise: How long does it take?
• Stages between dev-done and Go Live e.g:
Integration
Installing
Testing
Staging/Pre-Prod
Go Live
–
–
–
–
– Go Live
• Discuss in your team:
– How long for each stage in your environment?
– Which stage takes longest?
– Why does it take so long?
Intelligent Definition and Assurance
Slide 85

81.

Slow Release, Deployment and
Infrastructure Change
• We're interested in Continuous Delivery
• We've shifted left
– Testers get involved earlier
– Our requirements are better
– Devs write more tests and automate them
– Continuous Integration is our safety net
• We can deliver in days, but it still takes weeks
to deploy – what's going on?
Intelligent Definition and Assurance
Slide 86

82. Profiles of the three patterns

Deployment process 1
Does this sound familiar?
1. Integration – "nothing works"!
– First time developers code is merged it's clear
they haven't been talking to each other, or testing
2. Installing – "sometime, something somehow"
– A flaky, manual installation on new machines...
Doesn't work. Days or weeks to get it running.
Intelligent Definition and Assurance
Slide 87

83. Not three patterns; There are many

Deployment process 2
3. Testing – "it could be so much better!"
– Some functionality is available; but doesn't quite
match the requirements – which are changing
4. Staging/Pre-Prod – "why are we doing this?"
– Closer to production environment, but not exact;
flaky – in different ways; "you have 48 hours"
5. Go live – "just one more test..."
– Testers spend an afternoon 'just checking' before
the users are let loose
Intelligent Definition and Assurance
Slide 88

84. DevOps

Intersection of Dev, Ops and QA
Intelligent Definition and Assurance
Slide 89

85. Exercise: How long does it take?

Definition
"DevOps is a development methodology with a set
of practices aimed at bridging the gap between
Development and Operations, emphasizing
communication and collaboration, continuous
integration, quality assurance and delivery with
automated deployment”,Wikipedia
* Based on a review of definitions of DevOps from 238 papers
Intelligent Definition and Assurance
Slide 90

86. Slow Release, Deployment and Infrastructure Change

The ‘endless DevOps loop’
Why is it a
figure of eight?
Intelligent Definition and Assurance
Slide 91

87. Deployment process 1

DevOps - CAMS
• Culture
– Develop Dev-Ops relationship, communications,
collaboration, bring Dev closer to the customer
• Automation
– It's pervasive: builds, deployments, testing, monitoring, selfhealing, system rollouts, configuration...
• Metrics (Analytics)
– Experiment, capture, learn, improve both during
development and production
• Sharing
– Share ideas, and metrics; allow devs more control over
production systems using code.
Nice description of DevOps and CAMS here:
Intelligent Definition and Assurance
http://www.slideshare.net/geekle/devops-5348895/19-So_Whats_All_This_Changebr
Slide 95

88. Deployment process 2

The Tools Landscape

89. Intersection of Dev, Ops and QA

Periodic table of DevOps tools
https://xebialabs.com/periodic-table-of-devops-tools/
If only the world of DevOps
were so simple (and static)
Intelligent Definition and Assurance
Slide 97

90. Definition

DevOps tools landscape
Environments
Dev
Test
Production
CI
Service/API
Logging
Static Analysis
UI
Monitoring
E2E
Cust. Experience
Load/Perf
BI/Analytics
Build
DevTest
Integ. Test
Security
Platform
Virtualization
PaaS
IaaS
Deployment
Config/Provisioning
SCM
Containers
Repository
Release and Pipeline Orchestration
Collaboration/ChatOps
© 2016 Paul Gerrard, Gerrard Consulting

91. The ‘endless DevOps loop’

How many tools do you use?
NewRelic - Application monitoring - gives us the eyes on our app and how it's being used / performing
PaperTrail - Log file collector - brings in log files from various servers to one single place - great for systems running
across multiple servers
OpsView - Monitoring and alerting tool which we use to bring together monitoring from various systems
Nagios - Used for monitoring and alerting
PagerDuty - Used to alert (SMS and Email and Phone) when a service craps out
Elastic Search, Log Stash and Kibana - Data analysis and monitoring and trending - super powerful for analyzing
what our product is actually doing
Chef - Auto build and deploy technology to allow us to rapidly build and destroy environments (with Chef Kitchen
and Knife)
Vagrant - Create Virtual Environments
Real Time Board - Virtual Whiteboard - amazingly useful
Pivotal Tracker - Agile tracking tool
Fiddler - Proxy web tool
Firebug - Proxy web tool
Zed Attack Proxy - Security testing tool
Burpsuite - Security Testing Tool
HipChat - Real Time IM communication tool
Slack - Real Time IM Communication tool
Jira and Confluence - bug tracking and wiki
CloudFormations - Creates templates for Amazon instances
23
"No doubt we have some more hiding away but that's a pretty good list."
Intelligent Definition and Assurance
tools
Slide 99

92. What is DevOps?

The
Tools Knowledge Base
https://tkbase.com

93. Is DevOps about tools or people?

How many tools are there anyway?
• I'm researching tools for tkbase.com
– 2424 of which 686 are programming languages
– 1658 tools for DevOps, SDET & Testers
• Tool types and features
– https://tkbase.com/tools
• My guess is there are at least 2000.
Intelligent Definition and Assurance
Slide 105

94. So DevOps is just Dev and Ops working more closely?

314 bloggers, 33,034 blogposts indexed
and searchable (~40 new posts daily)
2,424 tools, indexed and searchable
Work in progress: tools tagged with
317 features
Intelligent Definition and Assurance
Slide 106

95. DevOps - CAMS

Platform
• IaaS, PaaS self-service
• Virtualization:
prepared images that
can be deployed to
provide full SaaS – if
required
• Containerization –
this is the challenge to
virtualization...
Intelligent Definition and Assurance
Slide 107

96. The Tools Landscape

Containerization
• Hottest topic in the DataCentre right now
• Virtual machines replicate much of the host machine
and kernel
• Containers reuse the host's kernel so are much lighter
and faster to boot
• One host can support hundreds, even thousands of
containers
• Some concerns and limitations especially in the area of
security and large-scale deployments (watch this space)
• Docker™ gets most attention, but there are
alternatives.
Intelligent Definition and Assurance
Slide 108

97. Periodic table of DevOps tools https://xebialabs.com/periodic-table-of-devops-tools/

Configuration/Provisioning
• "Infrastructure as code" is an emerging concept
– Define your environments as code in text files
– Use tools to follow those instructions
– Manage your configurations using SCM – just code
• Automated provisioning tools use the
configuration to create environments
• Some tools do everything e.g. Chef
• Some tools like Vagrant control the provisioners
(tools like Ansible, Puppet and Chef again).
Intelligent Definition and Assurance
Slide 109

98. DevOps tools landscape

Repository management
• Infrastructure as code means...
– Environments created accurately and reliably
– But these still take time
• Repositories hold environment 'components
we made earlier' to save even more time
• When you have a trusted recipe for an
environment or platform it can be stored,
managed and retrieved from the repository.
Intelligent Definition and Assurance
Slide 110

99. How many tools do you use?

Development
• Static analysis tools scan code and look for
problems and provide code metrics that indicate
'code smells'
• Build tools perform the source code downloads,
compilations and integration to create testable
apps or sub-systems
• DevTest tools are unit-test frameworks to test
components, classes and methods at a low level
• Integration tools run API based transactions to
demonstrate integration across components.
Intelligent Definition and Assurance
Slide 111

100. The Tools Knowledge Base

Test
• Service/API tests call web services/APIs directly
• GUI tools are the traditional user-oriented tests
that simulate realistic use-cases
• End-to-end tests harness both API and UI tests
to create end to end transactions
• Load/Performance tools simulate system loads
to explore behaviour and performance
• Security tools range from penetration tools to
instrumentation that can signal threats/break-ins.
Intelligent Definition and Assurance
Slide 112

101. Some Ancient History

Production
• Logging tools can be configured to
– Record execution statistics at virtually any level
– Trigger alarms for selected exceptions
• Monitoring tools range from the server,
database and network to application and security
• Customer experience (CX) tools provide
analytics on user journeys and behaviour to
support customer service and change-decisions
• Business Intelligence/Analytics tools provide
the analyses and management information based
on logs, instrumentation and monitoring.
Intelligent Definition and Assurance
Slide 113

102. CAST Report 4th edition (1999)

Release and pipeline
orchestration
• These tools define the deployment and release
pipeline, tasks and dependencies
• They can initiate the automated tasks in sequence
and provide a dashboard with progress
• Used collaboratively, everyone can see where the
release is at and what needs to be done to
resolve problems
• The data they capture on tasks, problems and
durations mean they can highlight bottlenecks
and opportunities for improvement.
Intelligent Definition and Assurance
Slide 114

103. Remember WinRunner?

Collaboration/ChatOps
• Tools so far enable technical folk to define and
implement environments, build, deploy and test
• Collaboration tools provide a focus for
distributed teams to communicate/collaborate
• ChatOps is collaboration using tools that are
integrated with DevOps
– ChatOps has been called 'conversations put to work'
– 'bots' report directly into chat rooms; bots can be
commanded directly through chat messages
• 'Conversation-driven development', online
decision-making and commitment.
Intelligent Definition and Assurance
Slide 115

104. CAST Report –tools hierarchy

ChatOps example
> bot: twitter says "we are down"
< user: @bot shut up twitter
> bot: twitter silenced for 15 min, get busy fixing stuff fast!
> bot: Nagios alert on web301: CRITICAL, high CPU over 95%
< user: @bot nagios ack web301
< user: @bot graph me io,net on web301
> bot: @user Here's your graph: https://mygraph.example.net/web301?show=io,net
> other_user: looks like it's just high load. Let's add couple of nodes!
< user: @bot autoscale add 2 nodes to cluster-3
> bot: @user On it, your execution is 5636fac02aa8856cc3f102ec
check the progress at https://st2.example.net/history/5636fac02aa8856cc3f102ec
Intelligent Definition and Assurance
Slide 116

105. How many tools are there anyway?

ChatOps

106.

Exercise: Create a Delivery Pipeline
from these activities (env/activity)
Developer/Run unit Tests
Production/Build Application in the Production environment
Developer/Build
Capacity/Build Application in the Capacity Environment
Acceptance/Build Application in Acceptance Environment
Capacity/Run Load Tests in the Capacity environment
Capacity/Smoke Test
Production/Perform Smoke Test in the Production environment
Developer/Compilation
Acceptance/Perform System Tests in Acceptance Environment
UAT/Smoke Test in the UAT Environment
Capacity/Deploy Environment
Production/Deploy the Production Environment
Acceptance/Smoke Test
UAT/Deploy Environment
Developer/Static Analysis
Acceptance/Deploy Acceptance Environment
UAT/Build Application in UAT Environment
Intelligent Definition and Assurance
Slide 118

107. Platform

Exercise: ChatOps
• Let's look at a simulation of a Deployment
Pipeline and use ChatOps to manage it
• http://chatopsdemo.slack.com credentials:
[email protected] password is 'p4ssw0rd‘
• Monitor the pipeline here:
http://dev.sp.qa/workshop/pipeline
Intelligent Definition and Assurance
Slide 119

108. Containerization

Test Automation Ambition and Reality

109. Configuration/Provisioning

"Automation is the future!"
• Heard that before?
• What exactly is possible and impossible with
automation, right here, right now?
• Where are the Digital, DevOps and Continuous
Delivery crazes leading us?
• Where do testers fit?
• How do testers work in high-paced, automationdominated environments?
• Let's look into the near future and discuss how
we survive or thrive.
Intelligent Definition and Assurance
Slide 121

110. Repository management

What Goes Wrong with
Test Execution
Automation?

111. Development

Tools are sensitive instruments
You don't need tools and tools won't help, if your software is unstable.
Intelligent Definition and Assurance
Slide 123

112. Test

The anti-regression goal
• What are you trying to do with automation?
• Anti-regression is the primary goal
– Detect unwanted change
– Insurance that developers can make changes safely
• How do you achieve this?
– The automation model must align with the
developer model
– You have to collaborate early to achieve this.
Intelligent Definition and Assurance
Slide 124

113. Production

Testing and automation
– a modelling problem
not a tools problem
Need to shift testing thinking to the left.
We model to improve requirements,
to eliminate mistakes in code and
automation.

114. Release and pipeline orchestration

Brian Marick's Original Agile
Testing Quadrant
Intelligent Definition and Assurance
Slide 126

115. Collaboration/ChatOps

Crispin & Gregory (Agile Testing version)
Intelligent Definition and Assurance
Slide 127

116. ChatOps example

Intelligent Definition and Assurance
Slide 128

117. ChatOps

Intelligent Definition and Assurance
Slide 129

118. Exercise: Create a Delivery Pipeline from these activities (env/activity)

What does this tell you?

119. Exercise: ChatOps

Intelligent Definition and Assurance
Slide 131

120. Test Automation - Ambition and Reality

Intelligent Definition and Assurance
Slide 132

121. "Automation is the future!"

How do we get better
at test automation?

122. What Goes Wrong with Test Execution Automation?

New Model Testing
where the checking
happens
The Left The Right
Intelligent Definition and Assurance
Slide 134

123. Tools are sensitive instruments

Left and Right
Thinking on the left
contributes to the definition
of features and how you test
them
Models on the left are the
blueprints for the tests &
automation on the right
The Left The Right
Your opportunity to
flush out muddled,
loose, incomplete,
flaky requirements
is on the left
Automation won’t
work if the thinking to
the left is not in place
Applying automation to muddled,
loose, incomplete, flaky features is
doomed before you start
If you aren’t involved on the left,
you aren’t in a good position to
test (with or without tools)
Intelligent Definition and Assurance
Slide 135

124. The anti-regression goal

Shift-Left (thinking, not people)
1. If you are not contributing on the left, you are
doomed to be a victim of muddled, incomplete
thinking
2. Challenge requirements thru example and
improve them
3. Share your models!
– Usage models: (test through the UI) must be
meaningful to users
– Service models: (services/APIs) meaningful to
developers and align with business rules
4. In case of test/automation ‘difficulty’ – go to 1.
Intelligent Definition and Assurance
Slide 136

125. Testing and automation – a modelling problem not a tools problem

Your homework for today
• Are tools just drones that can only execute
regression testing?
• Could tools find functional bugs in untested
software?
• If I can think of a test, can I automate the
This is where the tools play
application of it?
their part
• Think of tools as extensions
of your mind, not robots.
Intelligent Definition and Assurance
Slide 137

126. Brian Marick's Original Agile Testing Quadrant

Digital Testing Strategy
Agile Test Strategy – an Oxymoron?

127. Crispin & Gregory (Agile Testing version)

(Test Strategy as)
Agile Interventions
I’m using Scrum/Sprint terminology, but you don’t
have to of course
https://www.youtube.com/embed/Ed6YkYEkCRM

128.

Interventions (government client example)
Activity
Story Challenge
2
Story Definition
These activities are repeated
for each Sprint iteration
• On the following
slides, we highlight
8 interventions
• Some are test
phases, but some
aren’t
No.
1
7
8
9
3
Daily Stand-Up
4
Story Refinement
5
Developer Testing
6
Integration (and
incremental
System) Testing
System Testing
User Acceptance
Testing
Non-functional
Testing and PreProduction Testing
Intelligent Definition and Assurance
When?
As stories are added to the
Product Backlog
As stories are added to a
Sprint Backlog
Once per day during the
Sprint
Occurs throughout the Sprint
as new information emerges
Occurs throughout the Sprint
as the developer codes the
stories
During and at the end of
each sprint, including the
final sprint
At the end of each sprint,
including the final sprint
At the end of each sprint,
including the final sprint
Expected to take place on an
as-needs basis.
Slide 140

129.

1. Story Challenge
Suggest ‘what-ifs’ to
challenge new stories
and define story
headlines
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
Sprint Backlog
Sprint Backlog
Sprint Backlog
Sprint 2
Sprint 3
2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria
Sprint 1
Developed Stories
Developed Stories
Developed Stories
New Code
Integration into
Existing Code base
Automated testing
Increasing
Scope of
Sys. Test
and UAT
6. Integration Test
6. Integration Test
6. Integration Test
7. System Test
8. User Test
Increasing Scope of Integration,
System and Users Testing
Intelligent Definition and Assurance
Complete Tests after
Final Sprint
Slide 141

130. What does this tell you?

1. Story Challenge
Suggest ‘what-ifs’ to
challenge new stories
and define story
headlines
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
Sprint Backlog
Sprint Backlog
Sprint Backlog
Sprint 2
Sprint 3
2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria
Sprint 1
Developed Stories
Developed Stories
Developed Stories
New Code
Integration into
Existing Code base
Automated testing
Increasing
Scope of
Sys. Test
and UAT
6. Integration Test
6. Integration Test
6. Integration Test
7. System Test
8. User Test
Increasing Scope of Integration,
System and Users Testing
Intelligent Definition and Assurance
Complete Tests after
Final Sprint
Slide 142

131.

1. Story Challenge
Suggest ‘what-ifs’ to
challenge new stories
and define story
headlines
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
Sprint Backlog
Sprint Backlog
Sprint Backlog
Sprint 2
Sprint 3
2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria
Sprint 1
Developed Stories
Developed Stories
Developed Stories
New Code
Integration into
Existing Code base
Automated testing
Increasing
Scope of
Sys. Test
and UAT
6. Integration Test
6. Integration Test
6. Integration Test
7. System Test
8. User Test
Increasing Scope of Integration,
System and Users Testing
Intelligent Definition and Assurance
Complete Tests after
Final Sprint
Slide 143

132.

1. Story Challenge
Suggest ‘what-ifs’ to
challenge new stories
and define story
headlines
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
Sprint Backlog
Sprint Backlog
Sprint Backlog
Sprint 2
Sprint 3
2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria
Sprint 1
Developed Stories
Developed Stories
Developed Stories
New Code
Integration into
Existing Code base
Automated testing
Increasing
Scope of
Int. Sys.
and UAT
6. Integration Test
6. Integration Test
6. Integration Test
7. System Test
8. User Test
Increasing Scope of Integration,
System and Users Testing
Intelligent Definition and Assurance
Complete Tests after
Final Sprint
Slide 144

133. How do we get better at test automation?

1. Story Challenge
Suggest ‘what-ifs’ to
challenge new stories
and define story
headlines
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
Sprint Backlog
Sprint Backlog
Sprint Backlog
Sprint 2
Sprint 3
2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria
Sprint 1
Developed Stories
Developed Stories
Developed Stories
New Code
Integration into
Existing Code base
Automated testing
Increasing
Scope of
Int. Sys.
and UAT
6. Integration Test
6. Integration Test
6. Integration Test
7. System Test
8. User Test
Increasing Scope of Integration,
System and Users Testing
Intelligent Definition and Assurance
Complete Tests after
Final Sprint
Slide 145

134. New Model Testing

Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found,
stories tested, amended,
created
Daily Scrum
Stand-Up
Meeting
4. Story Refinement
Refine scenarios to
enhance story definition,
create system tests as
stories, as required
5) Developer Testing
Private ad-hoc tests and
build/run automated unit
tests
24
Hours
2-4 Weeks
Sprint Backlog
6) Integration/System Testing
Incorporate automated unit
tests into the CI regime.
On weekly basis and at end of
Sprint, deploy to System test
environment and tester runs
system tests.
Backlog tasks
expanded
by team
Product backlog
As prioritised by Product Owner
Intelligent Definition and Assurance
Potentially Shippable
Product increment
Slide 146

135. Left and Right

Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found,
stories tested, amended,
created
Daily Scrum
Stand-Up
Meeting
4. Story Refinement
Refine scenarios to
enhance story definition,
create system tests as
stories, as required
5) Developer Testing
Private ad-hoc tests and
build/run automated unit
tests
24
Hours
2-4 Weeks
Sprint Backlog
6) Integration/System Testing
Incorporate automated unit
tests into the CI regime.
On weekly basis and at end of
Sprint, deploy to System test
environment and tester runs
system tests.
Backlog tasks
expanded
by team
Product backlog
As prioritised by Product Owner
Intelligent Definition and Assurance
Potentially Shippable
Product increment
Slide 147

136. Shift-Left (thinking, not people)

Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found,
stories tested, amended,
created
Daily Scrum
Stand-Up
Meeting
4. Story Refinement
Refine scenarios to
enhance story definition,
create system tests as
stories, as required
5) Developer Testing
Private ad-hoc tests and
build/run automated unit
tests
24
Hours
2-4 Weeks
Sprint Backlog
6) Integration/System Testing
Incorporate automated unit
tests into the CI regime.
On weekly basis and at end of
Sprint, deploy to System test
environment and tester runs
system tests.
Backlog tasks
expanded
by team
Product backlog
As prioritised by Product Owner
Intelligent Definition and Assurance
Potentially Shippable
Product increment
Slide 148

137. Your homework for today

Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found,
stories tested, amended,
created
Daily Scrum
Stand-Up
Meeting
4. Story Refinement
Refine scenarios to
enhance story definition,
create system tests as
stories, as required
5) Developer Testing
Private ad-hoc tests and
build/run automated unit
tests
24
Hours
2-4 Weeks
Sprint Backlog
6) Integration/System Testing
Incorporate automated unit
tests into the CI regime.
On weekly basis and at end of
Sprint, deploy to System test
environment and tester runs
system tests.
Backlog tasks
expanded
by team
Product backlog
As prioritised by Product Owner
Intelligent Definition and Assurance
Potentially Shippable
Product increment
Slide 149

138. Digital Testing Strategy

The tester’s contribution
• Think of testing as interventions, not stages
• The testing role is redistributed and split
• The tester doesn’t own testing – think
TESTMASTER
• Identify the intervention points and negotiate
with your team
• But you must get involved early to trust the
test automation.
Intelligent Definition and Assurance
Slide 150

139. (Test Strategy as) Agile Interventions

Surviving Continuous
Delivery and DevOps
How do Testers fit into a DevOps
regime?

140. Interventions (government client example)

Tests, Automation and Trust
• Can we trust automation to do all our testing?
• Consider four areas:
1. Checks that can be automated by developers
2. Checks that can be automated (typically by system
testers) to exercise API-level, end-to-end
3. Compatibility checks across browsers, operating
systems, platforms.
4. Tests that can only be performed by a human
• Can you 'let go' of late, manual checking?
• It requires proactive effort and trust
Intelligent Definition and Assurance
Slide 152

141.

It's easier to trust if you're a
visionary
• Is your concern that you can't trust the
developers?
• Shifting test thinking to the left reduces doubt?
• Testers as pathfinders: identify/assess risks,
formulate tests; ensure they are included in
automation etc.
• You are not the gatekeeper of quality, the last
defence, the only person who cares
• Think more like a visionary, risk-identifier, risk
manager, pathfinder, a facilitator and a
coach/mentor.
Intelligent Definition and Assurance
Slide 153

142.

The Deployment Pipeline
• Consider your test and release processes as a
linked chain of processes (very waterfall hey?)
• Some processes are automated, some are
manual at least initially (and perhaps always)
• The transitions between them are decision
points (go back, recover, proceed to next etc.)
• Either the system or people make the
decisions, but people are always accountable.
Intelligent Definition and Assurance
Slide 154

143.

Test automation
• The manual interventions between automated
activities in DevOps are the pain points:
bottlenecks, delays, and error
• Manual testing is painful in this respect
– You love exploratory testing
– You think only you can find those gnarly bugs
– You think you are the only person trustworthy to
prevent disaster
• It might be painful for you to trust developers
and automation to do testing
• If it hurts, you must do it more often.
Intelligent Definition and Assurance
Slide 155

144.

Monitoring and Improvement
• A key discipline of DevOps is a deeper level of
monitoring in production
– Alerts on failure before users experience their impact
– Ambitious, but this is the ultimate goal
• When problems are encountered, use the analytics
– To trace the cause and resolve it
– To refine the test process ("let's adapt so we don't let that
bug through again")
• Automated tests in the process are 'monitoring' too
• DevOps process monitors enlarge the scope of testing.
Intelligent Definition and Assurance
Slide 156

145.

As a Tester, What Should You Do?
• There are 'critical moments' in all projects where
you might gather data and present feedback
• You need to identify your own critical moments
– Identify the choices that you and your team can make
– Offer your contribution and negotiate with your team
– These are your 'interventions'
• Offer leadership and guidance rather than to take
on responsibility for testing work
• It's easier to demonstrate your value to the team
if you take this approach – and the team won't
need as many testers.
Intelligent Definition and Assurance
Slide 157

146. Test Activities in the Sprint

I'm a Test Lead/Manager. What
Should I Do?
• It might be harder to justify your role if the
intent of management is to reduce the cost of
testing by shifting left
– Provide test and assurance skills to business
– Manage Requirements knowledge
– Be a TestMaster
– Be a DevOps- or ChatOps-Master
– Manage outsourced/offshore teams.
Intelligent Definition and Assurance
Slide 158

147. Test Activities in the Sprint

When should DevOps not be
attempted?
• A good question but the answer is a simple challenge:
– Why wouldn't you want developers and operations
people talking to each other?
– Why wouldn't you want more reliable builds and
deployments into test and production?
– Why wouldn't you want the best of technology to
support more accurate, efficient and informative pipelines?
• Testers need to embrace DevOps because it
provides opportunities to be pro-active, gain more
authority and respect in our project teams.
Intelligent Definition and Assurance
Slide 159

148. Test Activities in the Sprint

The phase after
development is
Re-Work

149. Test Activities in the Sprint

Paul Gerrard
email: [email protected]
Twitter: @paul_gerrard
LinkedIn: http://linkedin.com/in/paulgerrard
tkbase.com | testopera.com
@paul_gerrard
Thank-You!

150. The tester’s contribution

Testing, Analytics and
Decision-Making (ShiftRight)

151. Surviving Continuous Delivery and DevOps

Introduction
• The purpose of testing is to gather
information for someone to make a decision
• Testing provides the most valuable information
required by:
– Developers (to fix defects)
– Project managers (to understand and manage
progress)
– Stakeholders (to be updated and assured)
• In this one respect, testing is all-powerful.
Intelligent Definition and Assurance
Slide 163

152. Tests, Automation and Trust

SMAC – Real-Time Analytics
• Apps capture information about their users
and the usage patterns of their apps
themselves
– Apps collect data in real time
– Data is analyzed to detect trends, patterns of
behaviour, user preferences and opportunities for
improvement or new market initiatives
• All apps are instrumented to collect
information for decision making.
Intelligent Definition and Assurance
Slide 164

153. It's easier to trust if you're a visionary

Introducing Test Analytics
• “The capture, integration and analysis of test
and production monitoring data to inform
business and software development decisionmaking”
• Testers treat output from test management
tools as the main source of information for
reporting
• But this is far too limited a viewpoint.
Intelligent Definition and Assurance
Slide 165

154. The Deployment Pipeline

Modern Practices – Opportunities
for Testing
• Shift-Left aims to reduce, if not eliminate,
misunderstandings in requirements
• Pervasive automation in DevOps generates much
of the data we need automatically
• Results capture and analyses are no longer
manual; reporting is almost instant
• Some companies don’t log defects or bugs; when
defects are found – they are fixed
• But how does testing support decision-making?
Intelligent Definition and Assurance
Slide 166

155. Test automation

Testing and Decision-Making
• Testing supports decision making at all levels
• Testing Stakeholders can have many roles
– The information they require from testing varies
in line with their perspective and their need to
make decisions
– We must ask our Stakeholders what they would
find most useful, valuable and economic
• But how can we quantify testing?
• Time for some physics
Intelligent Definition and Assurance
Slide 167

156. Monitoring and Improvement

The challenge of test prediction
and planning
• The Testing Uncertainty Principle:
– We can predict test status, but not when it will be
achieved;
– We can predict when a test will end, but not its status
• We must treat exit criteria as planning
assumptions
• If we don’t meet the criteria
– Our assumptions are wrong
– Our plan is wrong
– We need to re-plan, or re-scope the project.
Intelligent Definition and Assurance
Slide 168

157. As a Tester, What Should You Do?

The Testing Theory of Relativity 1
• The value of a test is affected by many things
– A single test might cover five or five million lines of
code
– But who can say what that value is?
• We must rely on our Stakeholders to judge value
– They can’t put a value on any test – but they can say
which test is more valuable
• This sounds like a big disadvantage – but it isn’t
really.
Intelligent Definition and Assurance
Slide 169

158. I'm a Test Lead/Manager. What Should I Do?

The Testing Theory of Relativity 2
• Most of the challenge of test planning is scope
• We cannot test everything so we must prioritise
in some way
• Knowing the relative value of tests means we
should be able to select the more valuable ones
and set the other less valuable tests aside – for
now
• Given a valuable test, is a second test that does
the same thing as valuable? Not usually
• To discuss that, we need some quantum theory.
Intelligent Definition and Assurance
Slide 170

159. When should DevOps not be attempted?

Quantum Theory of Testing
• When we perform a test, pass or fail - the test
incrementally increases our knowledge and confidence
– Each test provides a quantum of evidence (yes/no,
true/false etc)
– We aggregate these to take a rounded view
– A test only has value if we learn something new
– If we learn little or nothing – the test has little or no value
• A test is significant if it increases our knowledge about
the system under test
• A repeat run of a test has no significance because we
learn nothing more.
Intelligent Definition and Assurance
Slide 171

160. The phase after development is Re-Work

Summary
• Test automation makes it easy to collect data on test outcomes;
how are we to take advantage of this opportunity?
• We need test models that are meaningful to our Stakeholders
– Stakeholder can recognise the value of our tests we propose
– As testers we can advise on the significance of these tests – the
coverage, if you like – with respect to the models themselves
• The discussion of value and significance helps us gauge the value of
our Test Automation
• For example, if a complete system test is scoped for automation…
– When first run the tests have value
– But when run repeatedly as regression tests, their significance and
therefore their value is much reduced
• The goal of “anti-regression” tests is better served by a smaller
number of tests that cover more – that is – they are significant.
Intelligent Definition and Assurance
Slide 172