Overview Verification and Validation for Concurrent Programs A State-of-Practice Questionnaire Margaret A. Wojcicki School of Information Technology and Electrical Engineering, University of Queensland e-mail: [email protected] Background Background Related Work Survey Methodology Survey Results Survey Analysis Conclusions and Future Work Background Verification: does a computer program conform to its specification? Validation: does a computer program meet the requirements of the client? Verification and validation (V&V): dynamic or static Cost-effectiveness factors: types of defects, cost of defect detection and false-positives (mostly static) Related Work Empirical studies provide information on V&V technologies Range from controlled experiments to surveys Survey provides practitioner input ‘real world’ answers to research questions Survey Methodology Survey Methodology Principal Results 1. V&V Australia (‘04) Online, telephone and interviews Low automation 2. V&V Sweden, US and Multinational (’03) Online Low automation Low unit testing 3. V&V Canada (‘04) N/A Low automation Low unit testing 4. V&V Sweden (’02) Workshop and interview sessions Larger companies more systematic and use commercial tools 1. 5. Software development NZ (‘00) Telephone and in person Larger companies more systematic 2. 6. Software Testing Institute (’02) Online Low automation Online sources of V&V technology information 7. Quality Assurance Institute (’02) At Intl. Conference on Software Testing Code reviews are most common static test Objectives: 1. 2. Examine current V&V technology use and context of use Examine decision process for V&V selection Survey sections: 3. 4. Context of V&V Technology Source of information Case Study Aspects Cost-effectiveness factors 1 Survey Methodology Survey Results - Demographics 35 responses Majority have at least 3 years of V&V experience Method: self-recruited +cross-sectional = non-systematic sampling Sample selection (convenience): 4. Survey Results – Programming Languages Number of Respondents 20 15 10 5 O th er Ac ad em ic 0 Positions Survey Results – Scope and Defects Predominantly Java, but also C++ (smaller numbers using Ada and C) average of 7.3 years experience with Java currently using version 1.5 or more recent Frequencies of Concurrency Defects Frequencies of Scope of V&V 35 35 30 30 N u m b er o f R esp o n d en t s 3. 25 So ft w ar e Te Q ste ua Ve lit r y rif A ic ss at ur io a n nc an e d V al S id of at tw io ar n e D ev el op Pr er oj ec tM an ag er R es ea rc he r 2. Concurrency-interest mailing list IBM developerWorks Multithreaded Java Programming forum Snowballing 5% response rate adequate (Lethbridge et al) N u m b er o f R esp o n d en ts 1. Respondent Positions 30 25 20 15 10 5 Survey Results – V&V 20 15 10 5 th e r Scope of V&V O m nc e ta ste Sy Ac ce p In te g ra tio Un n it 0 25 0 Deadlock Interference Starvation Other Concurrency Defects Survey Results – V&V V&V Technologies Applied 30 25 20 15 10 Other Development testing tools (JUnit) Model-checker Coverage Capture/Playback Automated static analysis tools 0 Pair programming 5 Code inspection/code walkthrough Number of Respondents 35 Code inspection/code walkthrough predominantly done ad-hoc and individually 60% use open source technology: mostly JUnit, FindBugs V&V Technologies 2 Survey Results – Decision Process 35 30 25 20 15 10 5 0 SD D N Research Literature Newgroups/Forums Marketing information Search Engine results on the V&V technology History of use in current company A SA History of use in other companies Number of Respondents Importance of Source of Information in Determining V&V Technology to Use Source of Information Number of Respondents Survey Results – Case Studies Importance of Case Study Factors for V&V Selection 35 30 25 20 15 10 5 0 NA SD D N A SA Case study Case study Case study Case study Case study in industry. with a large with with real in similar number of participants defects. application participants. applying domain. V&V technology. Case Study Factor Survey Results – Cost-effectiveness Survey Analysis – Needs Number of Respondents Ranking of Cost-Effectiveness Factors 20 18 16 14 12 10 8 6 4 2 0 Type of Defect False Positives Cost of using Tool High Medium Low V&V technology: Java 1.5, C++ Code inspection/code walkthrough (automation) Systematic inspections (groups) V&V technologies (coverage tools and modelcheckers) Rankings Survey Analysis - Needs Case studies: 1. 2. Real defects (bug repositories, benchmarks) Measurable results Decision Process: History in Industry Research Literature Survey Analysis Cost-effectiveness: Willing to deal with false-positives (quality reports) Also: code conversion, implementation/maintenance of tool, purchase cost relative to defects caught Previous surveys: Low Automation Unit testing versus acceptance (surveys 2 & 3) 3 Survey Analysis Threats to Validity: Low internal, but high external validity Generalise to software developers, but no control on background/knowledge Non-response bias (difficult to calculate) Convenience sampling Other issues: Definition of V&V (correctness) Sources of Information question biased to companies Ranking question (defaults to 1) Future Work Conclusions V&V mostly manual, aided with development testing tool automation (JUnit) Importance of real defects in case studies Many aspects of cost-effectiveness Qualitative information improves understanding of context Questions and Answers Examine locking mechanism used by practitioners Focus on performance issues rather than correctness and debugging tools In depth examination of C and C++ (memory management) Modification of TestCon and schema 4