Publications

Software and hardware security is a reality that all stakeholders must face, from hardware engineers to software developers to customers. As a direct result, the technology industry is facing a growing need for engineers who understand security principles at varying levels of abstraction. These engineers will need security-oriented perspectives stemming from both theoretical and practical disciplines, including software engineering, computer engineering, and computer science. Unfortunately, in traditional academic settings, secure software and hardware are typically taught independently despite being intertwined in practice. Consequently, the objective of this initiative is to prepare students to apply a security-oriented awareness to a broad range of hardware and software systems by developing a multi-disciplinary curriculum involving three departments. Our efforts at Rochester Institute of Technology focus on integrating security into software design and implementations, hardware design and implementations, and hardware-software co-design. In the cluster of courses described in this paper, we use cryptographic applications as the motivating security focus. We describe changes made to an existing introductory cryptography course, report on a recently-developed course entitled Hardware and Software Design for Cryptographic Applications, and present our plans for a Secure Software Engineering course.

With each new developer to a software development team comes a greater challenge to manage the communication, coordination, and knowledge transfer amongst teammates. Fred Brooks discusses this challenge in The Mythical Man-Month by arguing that rapid team expansion can lead to a complex team organization structure. While Brooks focuses on productivity loss as the negative outcome, poor product quality is also a substantial concern. But if team expansion is unavoidable, can any quality impacts be mitigated? Our objective is to guide software engineering managers by empirically analyzing the effects of team size, expansion, and structure on product quality. We performed an empirical, longitudinal case study of a large Cisco networking product over a five year history. Over that time, the team underwent periods of no expansion, steady expansion, and accelerated expansion. Using team-level metrics, we quantified characteristics of team expansion, including team size, expansion rate, expansion acceleration, and modularity with respect to department designations. We examined statistical correlations between our monthly team-level metrics and monthly product-level metrics. Our results indicate that increased team size and linear growth are correlated with later periods of better product quality. However, periods of accelerated team expansion are correlated with later periods of reduced software quality. Furthermore, our linear regression prediction model based on team metrics was able to predict the product's post-release failure rate within a 95% prediction interval for 38 out of 40 months. Our analysis provides insight for project managers into how the expansion of development teams can impact product quality.

Context: Researchers proposing a new metric have the burden of proof to demonstrate to the research community that the metric is acceptable in its intended use. This burden of proof is provided through the multi-faceted, scientific, and objective process of software metrics validation. Over the last 40 years, however, researchers have debated what constitutes a "valid" metric.
Aim: The debate over what constitutes a valid metric centers on software metrics validation criteria. The objective of this paper is to guide researchers in making sound contributions to the field of software engineering metrics by providing a practical summary of the metrics validation criteria found in the academic literature.
Method: We conducted a systematic literature review that began with 2,288 papers and ultimately focused on 20 papers. After extracting 47 unique validation criteria from these 20 papers, we performed a comparative analysis to explore the relationships amongst the criteria.
Results: Our 47 validation criteria represent a diverse view of what constitutes a valid metric. We present an analysis of the criteria’s categorization, relationships, advantages, and philosophical motivations behind the validation criteria. We then present a step-by-step process for selecting appropriate metrics validation criteria based on a metric’s intended use.
Conclusions: The diversity of motivations and philosophies behind the 47 validation criteria indicates that metrics validation is complex. Researchers proposing new metrics should consider the applicability of the validation criteria in terms of our categorization and analysis. Rather than arbitrarily choosing validation criteria for each metric, researchers should choose criteria that can confirm that the metric is appropriate for its intended use by inspecting the advantages that different criteria provide. We conclude that metrics validation criteria provide answers to questions that researchers have about the merits and limitations of a metric.

Security inspection and testing requires experts in security who think like an attacker. Security experts need to know code locations on which to focus their testing and inspection efforts. Since vulnerabilities are rare occurrences, locating vulnerable code locations can be a challenging task. We investigated whether software metrics obtained from source code and development history are discriminative and predictive of vulnerable code locations. If so, security experts can use this prediction to prioritize security inspection and testing efforts. The metrics we investigated fall into three categories: complexity, code churn, and developer activity metrics. We performed two empirical case studies on large, widely-used open source projects: the Mozilla Firefox web browser and the Red Hat Enterprise Linux kernel. The results indicate that 24 of the 28 metrics collected are discriminative of vulnerabilities for both projects. The models using all the three types of metrics together predicted over 80% of the known vulnerable files with less than 25% false positives for both projects. Compared to a random selection of files for inspection and testing, these models would have reduced the number of files and the number of lines of code to inspect or test by over 71% and 28%, respectively, for both projects.

Electronic health record (EHR) systems present a formidable "trustworthiness" challenge because people's health records, which are transmitted and protected by these systems, are just as valuable to a myriad of attackers as they are to health care practitioners. Major initiatives in EHR adoption and increased sharing of health information raise significant challenges for protecting the privacy of patients' health information. Dr. Laurie Williams created iTrust in 2005 as a course project for undergraduates in North Carolina State University's Software Engineering course. iTrust is intended as a patient-centric application for maintaining an EHR. An ideal health care system combines medical information from multiple sources to provide a summary or detail view of the history of a particular patient in a way that is useful to the health care practitioner. This chapter highlights the key pieces of iTrust's project artifacts that pertain to traceability and describes the project in detail.

Security inspection and testing requires experts in security who think like an attacker. Security experts need to know code locations on which to focus their testing and inspection efforts. Since vulnerabilities are rare occurrences, locating vulnerable code locations can be a challenging task. We investigated whether software metrics obtained from source code and development history are discriminative and predictive of vulnerable code locations. If so, security experts can use this prediction to prioritize security inspection and testing efforts. The metrics we investigated fall into three categories: complexity, code churn, and developer activity metrics. We performed two empirical case studies on large, widely-used open source projects: the Mozilla Firefox web browser and the Red Hat Enterprise Linux kernel. The results indicate that 24 of the 28 metrics collected are discriminative of vulnerabilities for both projects. The models using all the three types of metrics together predicted over 80% of the known vulnerable files with less than 25% false positives for both projects. Compared to a random selection of files for inspection and testing, these models would have reduced the number of files and the number of lines of code to inspect or test by over 71% and 28%, respectively, for both projects.

Open source software is often considered to be secure because large developer communities can be leveraged to find and fix security vulnerabilities. Eric Raymond states Linus' Law as "many eyes make all bugs shallow", reasoning that a diverse set of perspectives improves the quality of a software product. However, at what point does the multitude of developers become 'too many cooks in the kitchen', causing the system's security to suffer as a result? In a previous study, we quantified Linus' Law and 'too many cooks in the kitchen' with developer activity metrics and found a statistical association between these metrics and security vulnerabilities in the Linux kernel. In the replication study reported in this paper, we performed our analysis on two additional projects: the PHP programming language and the Wireshark network protocol analyzer. We also updated our Linux kernel case study with 18 additional months of newly-discovered vulnerabilities. In all three case studies, files changed by six developers or more were at least four times more likely to have a vulnerability than files changed by fewer than six developers. Furthermore, we found that our predictive models improved on average when combining data from multiple projects, indicating that models can be transferred from one project to another.

Understanding and measuring how groups of developers collaborate on software projects can provide valuable insight into software quality and the software development process. Current practices of measuring developer collaboration (e.g. with social network analysis) usually employ metrics based on version control change log data to determine who is working on which part of the system. Version control change logs, however, do not tell the whole story. Information about the collaborative problem-solving process is also documented in the issue tracking systems that record solutions to failures, feature requests, or other development tasks. To enrich the data gained from version control change logs, we propose two annotations to be used in issue tracking systems: solution originator and solution approver. We examined the online discussions of 602 issues from the OpenMRS healthcare web application, annotating which developers were the originators of the solution to the issue, or were the approvers of the solution. We used these annotations to augment the version control change logs and found 47 more contributors to the OpenMRS project than the original 40 found in the version control change logs. Applying social network analysis to the data, we found that central developers in a developer network have a high likelihood of being approvers. These results indicate that using our two issue tracking annotations identify project collaborators that version control change logs miss. However, in the absence of our annotations, developer network centrality can be used as an estimate of the project's solution approvers. This improvement in developer activity metrics provides a valuable connection between what we can measure in the project development artifacts and the team's problem-solving process.

Tracking organizations such as the US CERT show a continuing rise in security vulnerabilities in software, increasing awareness of insecure coding practices. Not all discovered vulnerabilities are equal - some have the potential to cause much more damage to organizations and individuals than others. In the inevitable absence of infinite resources, software development teams need to prioritize security fortification efforts to prevent the most damaging attacks. We propose the Protection Poker "game" as a collaborative means of guiding this prioritization. A case study of a Red Hat IT software maintenance team demonstrated the potential of Protection Poker for improving software security practices and team software security knowledge.

Open source software is often considered to be secure. One factor in this confidence in the security of open source software lies in leveraging large developer communities to find vulnerabilities in the code. Eric Raymond declares Linus' Law "Given enough eyeballs, all bugs are shallow." Does Linus' Law hold up ad infinitum? Or, can the multitude of developers become "too many cooks in the kitchen", causing the system's security to suffer as a result? In this study, we examine the security of an open source project in the context of developer collaboration. By analyzing version control logs, we quantified notions of Linus" Law as well as the "too many cooks in the kitchen" viewpoint into developer activity metrics. We performed an empirical case study by examining correlations between the known security vulnerabilities in the open source Red Hat Enterprise Linux 4 kernel and developer activity metrics. Files developed by otherwise- independent developer groups were more likely to have a vulnerability, supporting Linus' Law. However, files with changes from nine or more developers were 16 times more likely to have a vulnerability than files changed by fewer than nine developers, indicating that many developers changing code may have a detrimental effect on the system's security.

Discovery of security vulnerabilities is on the rise. As a result, software development teams must place a higher priority on preventing the injection of vulnerabilities in software as it is developed. Because the focus on software secu- rity has increased only recently, software development teams often do not have expertise in techniques for identifying security risk, understanding the impact of a vulnerability, or knowing the best mitigation strategy. We propose the Protection Poker activity as a collaborative and informal form of misuse case development and threat modeling that plays off the diversity of knowledge and perspective of the participants. An excellent outcome of Protection Poker is that security knowl- edge passed around the team. Students in an advanced undergraduate software engineering course at North Carolina State University participated in a Protection Poker session conducted as a laboratory exercise. Students actively shared misuse cases, threat models, and their limited software security expertise as they dis- cussed vulnerabilities in their course project. We observed students relating vul- nerabilities to the business impacts of the system. Protection Poker lead to a more effective software security learning experience than in prior semesters. A pilot of the use of Protection Poker with an industrial partner began in October 2008. The first security discussion structured via Protection Poker caused two requirements to be revised for added security fortification; led to the immediate identification of one vulnerability in the system; initiated a meeting on the prioritization of security defects; and instigated a call for an education session on preventing cross site scripting vulnerabilities.

Working remotely is becoming the norm for both professionals and students alike. Software development has become a global industry due to outsourcing, teleworking, flex time, and companies' desire to use the best and/or most economical talent regardless of where that talent is located. Professionals are not alone because students usually work from home despite having sufficient resources on campus. In this paper we share our experiences from using Jazz, a synchronous, collaborative development platform, with our inevitably distributed software engineering students. Eleven students optionally used the tool while working on a five-week team project. Students primarily used the version control, chat, and work items features in Jazz. We collected their reactions in retrospective essays and found that all Jazz students supported using Jazz in future semesters of the course. We also examined grade differences and found that the students who used Jazz were more successful than those who did not use Jazz.

Pair programming has proven to be a useful technique for developing high quality code while sharing knowledge throughout a team. Rapid global dispersion of software development teams, however, makes co-located pair programming a challenge, motivating the need for development tools tailored specifically for distributed pair programming. Previously, the Sangam Eclipse plug-in was developed to support distributed pair programming. More recently, the Jazz collaborative software development platform was built to support team communication and the sharing of life-cycle resources and to integrate a variety of disparate tools used by team members. We have ported Sangam to the Jazz platform to enable teams to pair program within their integrated team environment. In this paper, we describe Jazz Sangam, highlight the choices that lead to Sangam's current design, and discuss how Jazz Sangam can improve the distributed pair programming experience.

Software fails and fixing it is expensive. Research in failure prediction has been highly successful at modeling software failures. Few models, however, consider the key cause of failures in software: people. Understanding the structure of developer collaboration could explain a lot about the reliability of the final product. We examine this collaboration structure with the developer network derived from code churn information that can predict failures at the file level. We conducted a case study involving a mature Nortel networking product of over three million lines of code. Failure prediction models were developed using test and post-release failure data from two releases, then validated against a subsequent release. One model's prioritization revealed 58% of the failures in 20% of the files compared with the optimal prioritization that would have found 61% in 20% of the files, indicating that a significant correlation exists between file-based developer network metrics and failures.

Open-source project artifacts can be used to inject realism into software engineering courses or lessons on open-source software development. However, the use of open-source projects presents challenges for both educators and for students. Educators must search for projects that meet the constraints of their classes, and often must negotiate the scope and terms of the project with project managers. For students, many available open-source projects have a steep learning curve that inhibits them from making significant contributions to the project and benefiting from a "realistic" experience. To alleviate these problems and to encourage cross-institution collaboration, we have created the Repository for Open Software Education (ROSE) and have contributed three open-source projects intended for an undergraduate computer science or software engineering course. The projects in ROSE are education-friendly in terms of a manageable size and scope, and are intended to be evolved over many semesters. All projects have a set of artifacts covering all aspects of the development process, from requirements, design, code, and test. We invite other educators to contribute to ROSE and to use projects found on ROSE in their own courses.

Traditional approaches to semester-long projects in compiler courses force students to implement the early stages of a compiler in depth; since many students fall behind, they have little opportunity to implement the back end. Consequently, students have a deep knowledge of early material and no knowledge of latter material. We propose an approach based on incremental development and test-driven development; this approach solves the emphasis problem, provides experience with useful tools, and allows for such a course to be taught in a three or four weeks.