Return on Investment

153 METHODOLOGY FOR EVALUATION Defect Removal Model, Cost and Benefit Data, ,Return-on-Investment Model, , . and Costs and Benefits of Alternatives, which all lead up to a Cost and Benefit Model.Costs and benefits of various strategies is evaluated by interrelated techniques, starting with the Defect Removal Model. The Defect Removal Model, is a technique for evaluating models effectiveness, and once economic models employed it provides an empirically approach for comparing the costs and benefits of various models. existing cost and benefit data selected from the Literature Survey judiciously used for each of the analytical models. A Returnon-Investment (ROI) Model is based on the Defect Removal Model and provided by cost and benefit data, in order to des<;:ribe quality, productivity, cost, break even, and ROI estimates. Eventually, a Cost and Benefit Model is constructed from Cost and Benefit Criteria, The design of the Methodology is derived by McGibbon's (1996)[88], Jones' (1996 and 1997a), [73, 74] Grady's (1994 and 1997), [45, 47] and based on comparisons of SOPI costs and benefits. An analysis of costs and benefits by Herbsleb, Carleton, Rozrnn, Siegel, and Zubrow (1994), common method of cost and benefit study In [55] is the most use for the design of the Methodology. McGibbon's (1996) [88] study, .js used for two reasons, it is comprehensive in nature, and it exhi?its a broad range of comparative economic analyses between models. In addition, McGibbon's study provide detailed economic analyses associated with the Clean Room Methodology, Software Reuse, and even the Software Inspection Process. McGibbon's study establishes valid empiricaIlybased methodology for using existing cost and benefit data and analyses, for evaluating and selecting methods. 154 Cost and Benefit Criteria Three cost criteria and five benefit criteria for a total of 8 criteria are used to evaluate, -assess, and analyze various parameter: Training Hours, Training Cost, Effort, Cycle Time, Productivity, Quality, Return-on-Investment, and Break Even Hours. These criteria are used because of their usage and availability as explained by Table 29 Figure 12 Frequency df Metrics for 'Software Process Improvement (SI), and Table 46, Survey of Software Process Improvement (SPI) Costs and Benefits. Table 28 Survey of Metrics for Software Improvement Process showed 74 metric classes and 487 individual software metrics. However, Figure 12 showed Frequency of Metrics for Software Process Improvement (SPI), reclassified the 74 classes of 487 metrics into 11 classes: Productivity (22%), Design (18%), Quality (15%), Effort (14%), Cycle Time (9%), Size (8%), Cost (6%), Change (4%), Customer (2%), Performance (1 %), and Reuse {I %). This helped influence the selection of the eight criteria for costlbenefit analysis, since quantitative analyses is based on the existence and' present software metrics and measurement data available in published sources. These eight criteria are chosen because it is believed that these are the most meaningful indicators of both software process and software development process improvement performance, especially, Effort, Cycle Time, Productivity, Quality, Return-on-Investment, and Break Even Hours. Effort simply refers to cost, Cycle Time refers to duration, Productivity refers to number of units produced, Quality refers to number of defects removed, ROI refers to cost saved, and Break Even refers to length of time to achieve ROI. Organizations have chosen these software metrics to collect and report upon, doing exactly that over the years. 155 Quality software measurement data is a central part of this analysis and the direct basis for a return-on-investment (ROI) model that act as the foundation for computing ROI itself. Thus, the Quality criterion is an instrumental factor, and literature has abundantly and clearly reported Quality metric and measurement data, despite Quality's controversial and uncommon usage in management and measurement practice. The SEI reports that approximately 95.7% of software organizations are below CMM Level 4. CMM Level 4 is where software quality measurement is required. It is safe to assert that 95.7% of software organizations do not use or collect software quality measures. Strategies for Evaluation Seven models are chosen to evaluate, assess, and analyze cost and benefit the Personal Software Process (PSP), Clean Room Methodology, Defect Prevention Process, Software Inspection Process, Software Test Process, Capability Maturity Model, and ISO 9000 (Table 65). Table 65 Alternatives for Evaluating Costs and Benefits Categories Personal Software Process Type Product Appraisal Clean Room Methodology Formal Method Cost, Quality, ROi Defect Prevention Process Preventative Cost, Quality, ROi Software Inspection Process Product Appraisal Cost, Productivity, Quality, ROi Software Test Process Product Appraisal Cost, Quality Capability Maturity Model SPI Model ISO 9000 Quality Standard Typically Repclrted Data Cost. Productivity. Quality Cycle Time, Productivity, Quality Cost, Productivity, Quality Five models are relatively provide concise step-by-step guidance, Personal Software Process (PSP), Clean Room Methodology, Defect Prevention Process, Software Inspection Process, and Software Test Process. Two models are descriptive models that offer high level strategic and non-prescriptive guidance, Capability Maturity Model (CMM) and ISO 9000. 156 Two models are vertical life cycle methods offering complete, end-to-end processes for building software products, Personal Software Process (PSP), Clean Room Methodology, and three models' are vertical, process methods offering only partial software life cycle support. Defect Prevention Process, Software Inspection Process, and Software Test Process. All seven models are chosen for a number of reasons, maturity, completeness, acceptability, and most especially an abundance of quantitative cost and benefit data. The magnitude of the cost and benefit data (e.g., low implementation cost and high quality benefit). The Capability Maturity Model (CMM) and ISO 9000 are international standard and a certified international standard for SDPI and its related activity (e.g., software quality management). The Literature Survey proved instrumental in the identification and selection of th.ese seven models for cost and benefit analyses, particularly the Clean Room Methodology, because of their impressive costs and benefits. The Literature Survey also surfaced other alternatives, it is difficult to continued their analysed because of the lack of reported cost and benefit data, for Orthogonal Defect Classification (ODC), Product Line Management (PLM), and Software' Process Improvement and Capability dEtermination (SPICE). The seven models are principally aimed at improving quality, and may rightly be classified as software Engg. methodologies, It can be stated that no other method exists that report costs and benefits as impressive as the ones selected and examined here. While, broad an4 impressive studies do exist (McConnell, 1996), very little cost and benefit data is available to justify them. 157 Personal Software Process (PSP). The PSP is a relatively new software development life cycle, and software quality methodology (Humphrey, 1995). [58] The .PSP consists of five main software life cycle phases, Planning, HighLevel Design, High-Level Design Review, Development, and Post Implementation. The PSP is characterized by deliberate project planning and management, quantitative resource estimation and tracking, quality planning and management, highly structured individual reviews of software work products, and frequent software process and product measurements. Johnson and Disney (1998) [71] report that the PSP requires at least "12 separate paper forms, including a project plan summary, time recording log, process improvement proposal, size estimation template, time estimation template, object categories worksheet, test report template, task planning template, schedule planning template, design checklist, and code checklist." They stated that a single PSP project results in 500 individual software measurements, and that a small group of PSP projects easily results in tens of thousands of software measurements. The PSP is a highly prescriptive, step-by-step, measurementintensive software process for 'developing software with the explicit goal of improving software process performance, achieving SI, and resulting In measurably high quality software products. The PSP is small, efficient, tangibly examinable, and yields an abundance of data for research and analysis. Despite the newness of the PSP, and its popular but incorrect reputation as an academic software methodology, the PSP has a large amount of examinable data for research and analysis. The PSP is the smallest and most efficient method ever recorded, yielding the highest recorded ROI and 'productivity. Since the PSP is primarily conceived as a SPI method, the PSP is very inexpensive to operate, the PSP results in measurably high quality and productivity, and the PSP produces an abundance of measurement data & becomes a candidate for analysis and comparison. 158 Clean Room Methodology. The Clean Room Methodology, like the PSP, is a software development lifecycle and software quality methodology (Pressman, O 1997) [96] (Kaplan, Clark, and Tang, 1995) [81]. Clean Room consists of seven main software life cycle phases, Function Specification, Usage Specification, Incremental Development Plan, Formal Design and Correctness Specification, Random Test Case Generation, Statistical Testing, and Reliability Certification Model, according to Kaplan, Clark, and Tang. They further defines Clean Room as software life cycle phases Define Box Structures, Define Stimuli and Responses, Define State Boxes, Define Clear Boxes, Plan Statistical Reliability Certification, Define Usage Specification, Create Incremental Development Plan, and Develop Verifiable Designs. Pressman defines Clean Room as an eightphase software life cycle consisting of, Requirements Gathering, Statistical Test Planning, Box Structure Specification, Formal Design, Correctness Verification, Code Generation, Inspection, and Verification, Statistical Usage Testing, and Certification. Clean Room is characterized by rigorous requirements analysis, incremental development, formal specification and design, deliberate test planning, formal verification, rigorous testing. Clean Room is prescriptive, as it is improving software quality and resulting in ° high quality software products. Clean Room places little emphasis on process definition, performance, and measurement. In short Clean Room is a methods-based methodology used of basic mathematical proofs and verification. Clean Room has the goal of reducing the number of software defects committed, reducing reliance on the Software Inspection Process and Testing, and measurably increasing software quality levels not achieved by the Software Inspection Process. Clean Room is chosen because of software quality measurement data availability. Clean Room is also chosen for examination be~aus of a recent study of this methodology by McGibbon (1996) [88], exhibiting the costs and benefits of using Clean Room, and comparing with other methods chosen for examination in 159 this study, the Software Inspection Process. Clean Room is a technical, versus management, based methodology that relies on the use of basic formal specification and verification techniques. Clean Room has an reputation of being overly difficult, because it employs formal methods, and di,fficult to modem information technologies. It has the high cost of implementation and perception as being overly difficult to apply, Clean Room offers little evidence that it results in quality levels beyond those of much defined product appraisal techniques, such as the Software Inspection Process. McGibbon, defines that Clean Room results in smaller software source code sizes. Smaller software sizes naturally imply fewer opportunities to commit software defects, and subsequently, longterm efficiencies in software maintenance productivity. Defect Prevention Process. The Defect Prevention Process "is the process of improving quality and productivity by preventirtg the injection of defects into a product," according to Mays, Jones, Holloway, and Studinski (1990) [86]. According to Humphrey (\989) [57], "the fundamental objective of software defect prevention is to make sure that errors, once identified and addressed, do not occur again." Gilb and Graham (1993) [44] state that "the Defect Prevention Process is a set of practices that are integrated with the development process to reduce the number of errors developers actually make." Paulk, Weber, Curtis, and Chrissis (1996) [94] of the Software Engineering Institute (SEI) formally assert "the purpose of Defect Prevention is to identify the cause of defects and prevent them from recurring." The Defect Prevention Process defined by Jones (1985) [75] serves as a commonly accepted de facto standard, primarily consisting of five sub-processes, Stage Kickoff Meeting, Causal Analysis Meeting, Action Database, Action Team, and Repository. Defect Prevention is characterized by software defect data collection, defect classification, defect tracking, root-cause analysis, implementation of preventative actions, and most notably in-process education of commonly committed software defects. Defect Prevention has an explicit goal of increasing software quality and reliability. Defect Prevention is a method like 160 Clean Room, which has the explicit goal of reducing the number of software defects committed, reducing reliance on the Software Inspection Process and Testing. Defect Prevention relies heavily on rigorously collected and highly structured software defect data that is collected by less than 95% of all software organizations (Software Engineering Institute, 1999) [117], and thus is the limit of this approach. The importance of Defect Prevention is a critically strategic because there are reasons for Defect Prevention for examination and comparative analyses. Because, defect data is strategic Defect Prevention is strategic. Defect Prevention is well defined and there is a good amount of data available. Software Inspection Process. The Software Inspection Process is an in-process product appraisal activity, and is an instrumental component of contemporary software quality methodologies (Fagan, 1976) [35]. Inspections consist of six main sub-processes, Planning, Overview, Preparation, Inspection, Rework, and Followup. Inspections are characterized by highly structured team reviews by qualified peers, software defect identification, and software quality estimation , based on discovered software defects. More importantly, Inspections are characterized by concisely defIned, repeatable, and measurable processes, (Radice, Harding, Munnis, and Phillips, 1985) [99] (Radice, Roth, O'Hara, Jr., and Ciarfella, 1985) (lOO]. Inspections is considered as a quantitative software quality engineering methodologies (Kan, 1995; Humphrey, 1989, 1995, and 2000) [78, 58], such as Defect Prevention, CMM, software life cycle reliability modeling, software quality modeling, PSP, and the Team Software Process (TSP). Inspections are presented by software process metrics and measurements, Inspection is a highly prescriptive, step-by-step, measurement-intensive software process for validating software with the goal of improving' software process performance, resulting in high quality software products. While Inspections are even better defined and prescriptive than the PSP, Inspections only cover one aspect of software development, validation. 161 Inspections were selected for examination and comparative analysis because of the abundance of literature on Inspections, in both journals and textbooks, the abundance of reported and validated costs and benefits, and ability to develop ROI models and analyses based on Inspections. In fact, several key studies motivated the selection of Inspections for comparative analyses, McGibbon (1996), Grady (1997), Weller (1993), Russell (1991), and Rico (1996) [88, 45, 127, 110, 105]. McGibbon performed comparative analyses of Inspections, Clean Room, and Software Ryuse, showin& that Inspections exhibit an ROI beyond any other method Grady's [45] landmark study and comparison of methods, reporting that the use of Inspections have saved Hewlett Packard over $400 million. Weller and Russell explained that Inspections are extremely quantitative and effective, each responsible for helping change the image of Inspections from a qualitative Walkthrough-style technique to its real quantitative characterization. Rico (1993, 1996, and 1999) [104, 105, 107] showed how simple it is to examine the costs and benefits of Inspections, because of their measurability. Still, Inspections, like Clean Room and the PSP, has an reputation as being bureaucratic, too expensive, and too difficult to learn, with only marginal benefits. Software Test Process. The Software Test Process is a post-process product appraisal activity, According to IEEE (Std 1012-1986; Std 1059-1993) [67, 64], the Software Test Process consists of eight main sub-processes, Test Plan Generation, Test Design Generation, Test Case Generation, Test Procedure Generation, Component Testing, Integration Testing, System Testing, and Acceptance Testing. According to IEEE (J-Std 016-1995) [69), the Software Test process consists of seven main sub-processes, Test Planning, Test Environment Preparation, Unit Testing, Unit Integration Testing, Item Qualification Testing, Item Integration Testing, and System Qualification Testing. 162 According to IEEE (Std 12207.0-1996)[65], the Software Test process consists of six main sub-processes, Qualification Test Planning, Integration Test Planning, Unit Test Planning, Qualification Testing. Testing is Unit Testing, Integration Testing, and dynamic execution of software code and implementation based on predefined test procedures, by an independent test group other than the original programmer. According to Pressman (1997)[96] and Sommerville (1997)[118], Testing is described as white box (e.g., basis path and control structure) and black box (e.g., specification, interface, and operational) Testing techniques. Blackburn's (1998)[11) Testing approach is based on the theory that among the Testing techniques, boundary analysis is the most fruitful area for finding defects, and assert a direct correlation between the absence of boundary analysis defects and the overall absence of software product defects. Testing can be prescriptive, with a somewhat misguided, but honorable, goal of improving software quality. Testing is sometime misinterpreted for several important reasons, Testing doesn't involve estimating defect populations, little time is devoted to Testing, and Testing is usually conducted in an ad hoc and unstructured fashion, which all contribute to large latent defect population right into customer hands (Rico, 1999)[ 10 1]. However, there are some controversy to the strategic importance of Testing, as Lauesen and Younessi (1998)[83] claim that 55% of defects can only be found by Testing, while Weller (1993)[127] and Kan (1995)[78] describe that more than 98% of defects can be found before Testing begins. And, of course, Lauesen and Younessi explains the popular notion that software defect levels don't represent ultimate customer requirements, while Kan firmly shows a strong correlation between defect levels and customer satisfaction. Testing was chosen because there is an abundance of literature exhibiting the costs and benefits of Testing, 163 Thus, it now becomes imperative to examine various methods, quantify their individual costs and benefits, and direct SDPI resources to important areas yielding optimal ROI. Ironically, highly structured Testing is practiced by very few organizations, perhaps less than 95% (Software Engineering Institute, 1999). Capability Maturity Model for Software (CMM). The CMM is a software process improvement framework or reference model that emphasizes software quality (Paulk, Weber, Curtis, and Chrissis, 1995)[94]. The CMM is a framework or requirements organized by the following structural decomposition, Maturity Levels, Key Process Areas, Common Features, and Key Practices (Paulk, Weber, Curtis, and Chrissis). There are five Maturity Levels, Initial, Repeatable, Defined, Managed, and Optimizing (see Table 4 and Figure 20). Key Process Areas have Goals associated with them. There are 18 Key Process Areas divided among the Maturity Levels, zero for Initial, six for Repeatable, seven for Defined, two for Managed, and three for Optimizing. There are five Common Features associated with each of the 18 Key ·Process Areas" Commitment to Perform, Ability to Perform, Activities Performed, Measurement and Analysis, and Verifying Implementation. And, there are approximately 316 individual Key Practices or SDPI requirements divided among the 90 Common Features. The CMM is characterized by best practices for software development management, with a focus on software quality management. That is, the CMM identifies highpriority software management best practices, and their associated requirements. Thus, a software producing organization that follows the best practices prescribed by the CMM, and meets their requirements, is considered to have good software management practices. And, software-produqing organizations that don't meet the CMM's requirements are considered to have poor software management practices. 164 The CMM is a series of five stages or Maturity Levels of software management sophistication, Maturity Level One-Initial being ~orst and Maturity Level Five- Optimizing being considered best. At the first stage or Maturity Level, software management is asserted to be very unsophisticated or "immature" in CMM terminology. At the last or highest stage or Maturity Level, software management is asserted to be very sophisticated or "mature" in CMM terminology. The first or Initial Level has no SDPI requirements and characterizes poor software management practices. The second or Repeatable Level has six major best practices largely centered on software project planning and management. The third or Defined Level has seven major best practices centered on organizational SDPI management, process definition, and introduces some software quality management practices. The fourth or Managed Level only has two best practices emphasizing the use of software metrics and measurement to manage software development, as well as an emphasis on software quality metrics and measurement. The fifth and highest Optimizing Level focuses on Defect Prevention, the use of product technologies for process improvement, and carefully' managed process improvement. In essence, the CMM requires the definition of software project management practices, the definition of organizational software development practices, the measurement of organization process performance, and finally measurement-intensive process improvement. This is where the controversy enters the picture, while the CMM is reported to be prescriptive for SDPI, the CMM is not prescriptive for software project management, software development, or software measurement. The CMM merely identifies or names some important software processes, vaguely describes their characteristics, and even asserts a priority and order of SDPI focus (e.g., Maturity'Levels). Common issues are that the CMM doesn't identify all important software processes, doesn't group, prioritize, and sequence SDPI requirements appropriately, doesn't provide step-by-step prescriptive process definitions, and may actually effect SDPI by deferring soft~are quality measurement for several years. process and 165 Ironically, many commonly misperceive the CMM's 316 Key Practices to be the concise prescriptive requirements for software management and engineering. Fulfilling the CMM's 316 Key Practices will merely make an organization CMM-compliant. However, the CMM's 316 Key Practices neither fully describe .. functional software processes nor describe a best-in-class software life cycle like the PSP, TSP, or even the Clean Room Methodology do. In other words, the CMM attempts to describe the "essence" of best practices, but doesn't contain the detail necessary to define and use the recommended best practices by the CMM. One more time for emphasis, the CMM is not a software engineering life cycle standard like ISOIlEC 12207, EIAIlEEE 12207, or J-STD-016. The CMM is, international SDPI method and model, and there is an abundance of software measurement data associated with its use, particularly in the works of Herbsleb, Carleton, Rozum, Siegel, and Zubrow (1994)[55), Diaz and Sligo (1997)[31), and Haskell, Decker, and McGarry (1997)[53). While these studies have exhibited some rather impressive costs and benefits associated with using the CMM, it is not clear how many additional resources and techniques were required to meet the CMM's requirements. And, by actually meeting the CMM's requirements have actually been due to using other SPI methods such as the Software Inspection Process and the use of software defect density metrics, and attributing these successful outcomes to use of the CMM. ISO 900Q. According to Kan (1995)[58], "ISO 9000, a set of standards and guidelines for a quality assurance management system, represent another body of quality standards." According to NSF-ISR (199'9)[92], an international quality consulting firm, "ISO 9000 Standards were created to promote consistent quality practices across international borders and to facilitate the international exchange of goods and services." NSF - ISR goes on to assert that "meeting the stringent standards ofISO 9000 gives a company confidence in its quality management and assurance systems." 166 According the American Society for Quality Control (1999)[1], "The ISO 9000 series is a set of five individual, but related, international standards on quality management and quality assurance." The American Society for Quality Control states ISO 9000 standards "were developed to effectively document the quality system elements to be implemented in order to maintain an efficient quality system in your company." In short, ISO 9000 is a set of international standards for organizational quality assurance (QA) standards, systems, processes, practices, and procedures. ISO 9000-3, Qucility Management and Quality Assurance Standards- Part 3: Guidelines for the Application of ISO 9001 to the Development, Supply, and Maintenance of Software, specifies 20 broad classes of requirements or elements. The first 10 ISO 9000-3 quality system elements are, Management Responsibility, Quality System, Contract Review, Design Control, Document Control, Purchasing, Purchaser-Supplied Product, Product Identification and Traceability, Process Control, and Inspection and Testing. The last 10 ISO 9000-3 quality system elements are, Inspection, Measuring, and Test Equipment, Inspection and Test Status, Control of Nonconforming Product, Corrective Action, Handling, Storage, Packaging; and Delivry~ Quality Records, Internal Quality Audits, Training, Servicing, and Statistical Techniques. ISO 9000 is characterized by the creation, existence, and auditing of a "quality manual" that "aids in implementing your quality system; communicates policy, procedures and requirements; outlines goals and structures of the quality system and ensures compliance," ISO 9000 describes the essence of an organizational quality management system at the highest levels, much like the CMM, and it is descriptive, but not very prescriptive. While, other models like the PSP, TSP, Clean Room, and Inspections reqUIre actual conformance to step-by-step software quality methodology, ISO 9000 was chosen for examination and comparative analyses as a model , because as Kaplan, Clark, and Tang (1995)[8\] state, "the whole world was rushing to adopt ISO 9000 as a quality standard." Studies by Kaplan, Clark, 167 and Tang and Haskell, Decker, and McGarry (1997)[53] were instrumental keys to identify the costs and benefits ofISO 9000 as ,a Engg. Mod<;1 .There are many authoritative surveys have been conducted measuring international organization who are using ISO 9000-compliant quality management systems. According to Arditti (1999)[2], Lloyd Register reports survey respondent perceptions such as, improved management-86%, better customer service-73%, improved efficiency and productivity-69%, reduced waste-53%, improved staff motivation and reduced stafftumover-50%, and reduced costs-40%. Irwin Publishing reports respondent perceptions such as, higher quality-83%, competitive advantage-69%, less customer quality audits-50%, and increased customer demand-30%, as per Garver (1999)[43],'s survey, Improved Management Control-83%, Improved Customer Satisfaction 82%, Motivated Workforce-61 %, Increased Opportunity To Win Work-62%, Increased Productivity Efficiency-60%, Reduced Waste60%, More Effective Marketing-52%, Reduced Costs-50%, and Increased Market Share-49%. While these statistics provide good figures , these statistics do not explain "actual" benefits, but merely "perceived" ones. It' s not clear how much actual benefits arise from using ISO 9000-compliant quality management systems. However, as mentioned before Kaplan, Clark, and Tang and Haskell, Decker, and McGarry, among others, have provided enough quantitative cost and benefit information to include the use of ISO, 9000 as a ipodel , given its tremendous popularity, and perceived ubiquity. It is important to note that very few firms actually employ ISO 9000, at least domestically. Defect Removal Model The defect removal model is a tool for managing software quality as software products are developed, by evaluating phase-by-phase software defect removal efficiency (Kan, 1995)[78]. The defect removal model has historically been used to model software process, software project management, and software verification and validation effectiveness (Sulack, Lindner, and Dietz, 1989; Humphrey, 1989 and 1995; Gilb, 1993; Kan, 1995; McGibbon,' 1996; Ferguson, 168 Humphrey, Khajenoori, Macke, and 1997; Matvya, Rico, 1999)[118,57,58,44,78,88,37,107]. Software Development Process Improvement (SDPI) costs and benefits, particularly return-an-investment (RGI), are also modeled by the defect removal model (Gilb, 1993; Grady, 1994 and 1997; McGibbon, 1996)[44,47,45,88]. The defect removal model is similarly represented by the dynamics of the Rayleigh life cycle reliability model. The notion being that software defects should be eliminated early in software life cycles, and that the economics of late defect elimination are cost-prohibitive. According to Kan, "the phase-based defect removal model summanzes the interrelations among three metrics-defect injection, defect removal, and effectiveness." arithmetically "defects at the exit of a development setup - defects escaped from previous setup + defects injected in current setup - defects removed in current setup." Kan states that the defect removal model is a good " : ~ tool for software quality management, not software reliability modeling and estimation. Rico's (1999)[ I 07] ~asic defect removal model, comparing the costs and benefits of the Personal Software Process (PSP), Software Inspection Process, and Software Test Process, is expanded upon to establish the guide line for the analysis. The defect removal model is used to design and construct an empirically valid ROI model, as well as the empirical framework for evaluating the costs ·and benefits of the tafl~edmo .This study evaluates the costs and benefits of the PSP, Clean Room, Prevention, Inspection, Test, CMM, and ISO 9000. a particular ROI or break even point is asserted for the PSP, Inspection, or , Test, in which these leads to a critically strategic comparative factors, then a empirical foundation is used to established and validate these assertions. 169 Table 66 Humllhrex's Defect Removal Model 1571 Defects Residual Injected RemDved Remaining Injection Rate Removal Efficiency Cumulative Efficiency Inspection Derects Development Defects Inspection Eflicen~ Code High Detailed Level Level Un it Design 0 10 6 Design Test 34 5 4 13 21 12 4 13 10 60°/. 60'% 6 21 63 42 34 63 480/0 58% 55°/. 64% 18 18 60 6 60 Jntegration System U,age Test Te,t 13 2 6 9 51°/. 19 2 8 13 2 38% 2 40-/. 12 0 3 100% 81% 87% 91% 100% 80 88 20 19 5 9 3 94 63.8% Humphrey. One of the first defect removal models was presented by Humphrey (1989)[57] to explain defect removal efficiency, is his book on the SEl's CMM (see Table 66). He presented seven software life cycle phases, Detailed Level Design, Code, Unit Test, Integration Test, System Test, and Usage, for software quality analysis on a stage-by-stage basis. He also used 10 software measurements for in-process software quality analysis. Residual defects are the estimated starting defects at the beginning of each stage. Injected is the number of new defects committed in each stage. Removed is the number of defects eliminated in. each stage. Remaining is injected less removed defects. Injected Rate is the same as Injected defects in this example. Removal Efficiency is a ratio of Removed to Residual and Injected defects. Cumulative Efficiency is a ratio of all Removed to all Residual and Injected defects. Inspection Defects are software defects found by the Software Inspection Process. Development Defects are software defects committed before product delivery. And Inspection Efficiency is an estimate of the overall Software Inspection Process effectiveness, or ratio of software defects found by , Inspections to total estimated defects. This is a realistic model because it presents modest Inspection efficiencies, and stage-by-stage software defect injection. 170 Table 61 PSP Cost/Benefits Inspt£.tion ,Test Program Size 10 KSLOC 10 KSLOC 10 KSLOC StOft Defect. 1,000 97.24 1,000 708 1,000 nla 666 900 nla 6.85 1.27 nla Review Hours Review Defects Defects per Hour 333 100 1,000 Test Hours 60.92 1,144 11,439 T ••t Def••" 333 90 900 12.71 t2.71 Total Houl'll 5.47 400 • 1,852 11,439 Total Def.cts 1,000 990 900 Quality Benefit 100X lOX n/. 0 10 100 29X 6X nla Start Defects Defects per Hour Delivered Defeots Cost Benefit Rico. The defect removal model in Table 67 was created by Rico (1999)[107J, and is a hybrid of models by .Ferguson, Humphrey, Khajenoori, Macke, and Matvya (1997)[37J, and McGibbon (1996)[88J. Program Size is thousands of source lines of code. Start Defects are the estimated software defects. Review Hours are the number of static analysis hours. Review Defects are the number of defects found by static analysis. Defects per Hour are the ratio of Review Defects to Review Hours. Start Defects are the number of defects escaping reviews. Test Hours are the number of dynamic analysis hours. Test Defects are the number of defects found by dynamic analysis. Defects per }lour are the ratio of Test Defects . , to Test Hours. Total Hours are the sum of Review Hours and Test Hours. Total Defects are the sum of Review. Defects and Test Defects. Quality Benefit is a ratio of poorest Delivered Defects to next best Delivered Defects. Delivered Defects are the numbers of defects escaping static and dynamic analysis. Cost Benefit is a ratio of poorest Total Hours to next best Total Hours analysis. This software quality-based ROI model is an of an earlier work by Rico (1999)[ 107) that was designed for the express purpose of evaluating ROl. It is a seemingly simple, though intricately complex composite of mUltiple sub-models, simulating the effects of several methods on efficjency, product~viy, quality.