presentation

Overview
Verification and
Validation for Concurrent
Programs
A State-of-Practice Questionnaire
Margaret A. Wojcicki
School of Information Technology and Electrical
Engineering, University of Queensland
e-mail: [email protected]
Background
Background
Related Work
Survey Methodology
Survey Results
Survey Analysis
Conclusions and Future Work
Background
Verification: does a computer program conform
to its specification?
Validation: does a computer program meet the
requirements of the client?
Verification and validation (V&V): dynamic or
static
Cost-effectiveness factors: types of defects, cost
of defect detection and false-positives (mostly
static)
Related Work
Empirical studies provide information on
V&V technologies
Range from controlled experiments to
surveys
Survey provides practitioner input
‘real world’ answers to research questions
Survey Methodology
Survey
Methodology
Principal Results
1. V&V Australia
(‘04)
Online, telephone
and interviews
Low automation
2. V&V Sweden,
US and
Multinational (’03)
Online
Low automation
Low unit testing
3. V&V Canada
(‘04)
N/A
Low automation
Low unit testing
4. V&V Sweden
(’02)
Workshop and
interview sessions
Larger companies more systematic and use commercial
tools
1.
5. Software
development NZ
(‘00)
Telephone and in
person
Larger companies more systematic
2.
6. Software Testing
Institute (’02)
Online
Low automation
Online sources of V&V technology information
7. Quality
Assurance Institute
(’02)
At Intl. Conference
on Software Testing
Code reviews are most common static test
Objectives:
1.
2.
Examine current V&V technology use and context of
use
Examine decision process for V&V selection
Survey sections:
3.
4.
Context of V&V Technology
Source of information
Case Study Aspects
Cost-effectiveness factors
1
Survey Methodology
Survey Results - Demographics
35 responses
Majority have at least 3 years of V&V experience
Method:
self-recruited +cross-sectional =
non-systematic sampling
Sample selection (convenience):
4.
Survey Results –
Programming Languages
Number of Respondents
20
15
10
5
O
th
er
Ac
ad
em
ic
0
Positions
Survey Results –
Scope and Defects
Predominantly Java, but also C++ (smaller
numbers using Ada and C)
average of 7.3 years experience with Java
currently using version 1.5 or more recent
Frequencies of Concurrency Defects
Frequencies of Scope of V&V
35
35
30
30
N u m b er o f R esp o n d en t s
3.
25
So
ft
w
ar
e
Te
Q
ste
ua
Ve
lit
r
y
rif
A
ic
ss
at
ur
io
a
n
nc
an
e
d
V
al
S
id
of
at
tw
io
ar
n
e
D
ev
el
op
Pr
er
oj
ec
tM
an
ag
er
R
es
ea
rc
he
r
2.
Concurrency-interest mailing list
IBM developerWorks Multithreaded Java
Programming forum
Snowballing
5% response rate adequate
(Lethbridge et al)
N u m b er o f R esp o n d en ts
1.
Respondent Positions
30
25
20
15
10
5
Survey Results – V&V
20
15
10
5
th
e
r
Scope of V&V
O
m
nc
e
ta
ste
Sy
Ac
ce
p
In
te
g
ra
tio
Un
n
it
0
25
0
Deadlock
Interference
Starvation
Other
Concurrency Defects
Survey Results – V&V
V&V Technologies Applied
30
25
20
15
10
Other
Development
testing tools
(JUnit)
Model-checker
Coverage
Capture/Playback
Automated static
analysis tools
0
Pair programming
5
Code
inspection/code
walkthrough
Number of Respondents
35
Code inspection/code walkthrough
predominantly done ad-hoc and
individually
60% use open source technology: mostly
JUnit, FindBugs
V&V Technologies
2
Survey Results –
Decision Process
35
30
25
20
15
10
5
0
SD
D
N
Research
Literature
Newgroups/Forums
Marketing
information
Search Engine
results on the V&V
technology
History of use in
current company
A
SA
History of use in
other companies
Number of Respondents
Importance of Source of Information in
Determining V&V Technology to Use
Source of Information
Number of Respondents
Survey Results – Case Studies
Importance of Case Study Factors for V&V
Selection
35
30
25
20
15
10
5
0
NA
SD
D
N
A
SA
Case study Case study Case study Case study Case study
in industry. with a large
with
with real
in similar
number of participants defects.
application
participants. applying
domain.
V&V
technology.
Case Study Factor
Survey Results –
Cost-effectiveness
Survey Analysis – Needs
Number of Respondents
Ranking of Cost-Effectiveness Factors
20
18
16
14
12
10
8
6
4
2
0
Type of Defect
False Positives
Cost of using Tool
High
Medium
Low
V&V technology:
Java 1.5, C++
Code inspection/code walkthrough
(automation)
Systematic inspections (groups)
V&V technologies (coverage tools and modelcheckers)
Rankings
Survey Analysis - Needs
Case studies:
1.
2.
Real defects (bug repositories, benchmarks)
Measurable results
Decision Process:
History in Industry Research Literature
Survey Analysis
Cost-effectiveness:
Willing to deal with false-positives (quality reports)
Also: code conversion, implementation/maintenance
of tool, purchase cost relative to defects caught
Previous surveys:
Low Automation
Unit testing versus acceptance (surveys 2 & 3)
3
Survey Analysis
Threats to Validity:
Low internal, but high external validity
Generalise to software developers, but no control on
background/knowledge
Non-response bias (difficult to calculate)
Convenience sampling
Other issues:
Definition of V&V (correctness)
Sources of Information question biased to companies
Ranking question (defaults to 1)
Future Work
Conclusions
V&V mostly manual, aided with
development testing tool automation
(JUnit)
Importance of real defects in case studies
Many aspects of cost-effectiveness
Qualitative information improves
understanding of context
Questions and Answers
Examine locking mechanism used by
practitioners
Focus on performance issues rather than
correctness and debugging tools
In depth examination of C and C++ (memory
management)
Modification of TestCon and schema
4