Andy Meneely
Assistant Professor, Software Engineering

Department of Software Engineering
Rochester Institute of Technology
134 Lomb Memorial Drive
Rochester, NY 14623-5608

Golisano 70-1547
585-475-7829
andy (AT) se.rit.edu

Interests

Empirical software engineering, software security, collaborative development, open source development, socio-technical team aspects, metrics and measurement, applied machine learning and data mining

Office hours

See myCourses for up-to-date office hours.

Bio

Andy is just a guy who loves computing. He's been an assistant professor of Software Engineering at RIT since 2011. Prior to then, Andy got his PhD in Computer Science at North Carolina State University in Raleigh, North Carolina under Laurie Williams. His doctoral dissertation, entitled Investigating the Relationship between Developer Collaboration and Software Security involved formulating metrics to examine the socio-technical structure of software development teams using social network analysis. His research has resulted in many top-tier academic publications. He also earned his Masters at NCSU in 2008. Andy got his Bachelors at Calvin College, Grand Rapids, MI where he was a double-major in Computer Science and Mathematics.

Recent Publications

Word cloud visualization of the paper When a patch goes bad: Exploring the properties of vulnerability-contributing commits Andrew Meneely, Harshavardhan Srinivasan, Ayemi Musa, Alberto Rodríguez Tejeda, Matthew Mokary & Brian Spates Proceedings of the 2013 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement to appear 2013
Security is a harsh reality for software teams today. Developers must engineer secure software by preventing vulnerabilities, which are design and coding mistakes that have security consequences. Even in open source projects, vulnerable source code can remain unnoticed for years. In this paper, we traced 68 vulnerabilities in the Apache HTTP server back to the version control commits that contributed the vulnerable code originally. We manually found 124 Vulnerability-Contributing Commits (VCCs), spanning 17 years. In this exploratory study, we analyzed these VCCs quantitatively and qualitatively with the over-arching question: “What could developers have looked for to identify security concerns in this commit?” Specifically, we examined the size of the commit via code churn metrics, the amount developers overwrite each others’ code via interactive churn metrics, exposure time between VCC and fix, and dissemination of the VCC to the development community via release notes and voting mechanisms. Our results show that VCCs are large: more than twice as much code churn on average than non-VCCs, even when normalized against lines of code. Furthermore, a commit was twice as likely to be a VCC when the author was a new developer to the source code. The insight from this study can help developers understand how vulnerabilities originate in a system so that security-related mistakes can be prevented or caught in the future.
 @article{MeneelyESEM2013,
  title = {When a patch goes bad: Exploring the properties of vulnerability-contributing commits},
  booktitle = {Proceedings of the 2013 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement},
  series = {ESEM '10},
  year = {2013},
  location = {Baltimore, MD, USA},
  pages = {to appear},
  numpages = {10},
  author = {Meneely, Andrew and Srinivasan, Harshavardhan and Musa, Ayemi and Tejeda, Alberto Rodríguez and Mokary, Matthew and Spates, Brian},
  keywords = {vulnerability, churn, socio-technical, empirical},
  abstract = {Security is a harsh reality for software teams today. Developers must engineer secure software by preventing vulnerabilities, which are design and coding mistakes that have security consequences. Even in open source projects, vulnerable source code can remain unnoticed for years. In this paper, we traced 68 vulnerabilities in the Apache HTTP server back to the version control commits that contributed the vulnerable code originally. We manually found 124 Vulnerability-Contributing Commits (VCCs), spanning 17 years. In this exploratory study, we analyzed these VCCs quantitatively and qualitatively with the over-arching question: “What could developers have looked for to identify security concerns in this commit?” Specifically, we examined the size of the commit via code churn metrics, the amount developers overwrite each others’ code via interactive churn metrics, exposure time between VCC and fix, and dissemination of the VCC to the development community via release notes and voting mechanisms. Our results show that VCCs are large: more than twice as much code churn on average than non-VCCs, even when normalized against lines of code. Furthermore, a commit was twice as likely to be a VCC when the author was a new developer to the source code. The insight from this study can help developers understand how vulnerabilities originate in a system so that security-related mistakes can be prevented or caught in the future.}
}
 

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper WIP: Teaching web engineering using a project component Daniel E. Krutz & Andrew Meneely Proceedings of the 2013 Frontiers in Education Conference to appear 2013
Web applications are an intricate part of today’s world. Everything from banking to checking our Facebook status may now be done through the use of web applications. The pressure of creating secure web applications on time and on budget is felt by innumerable software developers. Late or buggy software may have adverse financial implications while an insecure application could expose the private information of countless individuals. Today’s students need to balance numerous concerns in order to create a web application that is robust, on time and on budget. These software engineering techniques are amenable to web applications due to their changing requirements, diverse development teams and iterative nature. Unfortunately, the majority of students lack these skills upon graduation. Web engineering is the utilization of software engineering techniques to web technologies. While Web engineering has recently begun to emerge as its own field of engineering, it still lacks the maturity of legacy software engineering fields. At the Department of Software Engineering at the Rochester Institute of Technology, we created a course called Web Engineering. As part of this course, we developed an innovative project component which focused on students following software engineering development principles such as elicitation, requirements generation, testing and deployment. Some of the requirements for this project included the creation of a custom calendar, several tie-ins with the Facebook API and creation of custom web services. An additional differentiating factor for this project was the cross course collaboration with a simultaneous security class. As part of this partnership between the two classes, students from the security course would routinely examine the web engineering project. Security students provided feedback to the web engineering students, and the web engineering students were expected to react. To our knowledge, no other curriculum uses such a project driven approach to emphasis engineering techniques. Furthermore we encourage security through the collaboration with the security course in our department. The student reaction to this course has been overwhelmingly positive. The initial offering of this course was in the Spring of 2012 with a second iteration in the Spring of 2013. This experience report will describe the project, general course structure as well as various obstacles and lessons learned.
 @article{KrutzFIE2013,
  title = {WIP: Teaching web engineering using a project component},
  booktitle = {Proceedings of the 2013 Frontiers in Education Conference},
  series = {FIE '13},
  year = {2013},
  pages = {to appear},
  numpages = {3},
  author = {Krutz, Daniel E. and Meneely, Andrew},
  keywords = {web, engineering, education, security},
  abstract = {Web applications are an intricate part of today’s world. Everything from banking to checking our Facebook status may now be done through the use of web applications. The pressure of creating secure web applications on time and on budget is felt by innumerable software developers. Late or buggy software may have adverse financial implications while an insecure application could expose the private information of countless individuals. Today’s students need to balance numerous concerns in order to create a web application that is robust, on time and on budget. These software engineering techniques are amenable to web applications due to their changing requirements, diverse development teams and iterative nature. Unfortunately, the majority of students lack these skills upon graduation. Web engineering is the utilization of software engineering techniques to web technologies. While Web engineering has recently begun to emerge as its own field of engineering, it still lacks the maturity of legacy software engineering fields.  At the Department of Software Engineering at the Rochester Institute of Technology, we created a course called Web Engineering. As part of this course, we developed an innovative project component which focused on students following software engineering development principles such as elicitation, requirements generation, testing and deployment. Some of the requirements for this project included the creation of a custom calendar, several tie-ins with the Facebook API and creation of custom web services. An additional differentiating factor for this project was the cross course collaboration with a simultaneous security class. As part of this partnership between the two classes, students from the security course would routinely examine the web engineering project. Security students provided feedback to the web engineering students, and the web engineering students were expected to react.  To our knowledge, no other curriculum uses such a project driven approach to emphasis engineering techniques.  Furthermore we encourage security through the collaboration with the security course in our department. The student reaction to this course has been overwhelmingly positive. The initial offering of this course was in the Spring of 2012 with a second iteration in the Spring of 2013. This experience report will describe the project, general course structure as well as various obstacles and lessons learned.}
}
 

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper Vulnerability of the day: concrete demonstrations for software engineering undergraduates Andrew Meneely & Samuel Lucidi Proceedings of the 2013 International Conference on Software Engineering 1154–1157 2013
Software security is a tough reality that affects the many facets of our modern, digital world. The pressure to produce secure software is felt particularly strongly by software engineers. Todays software engineering students will need to deal with software security in their profession. However, these students will also not be security experts, rather, they need to balance security concerns with the myriad of other draws of their attention, such as reliability, performance, and delivering the product on-time and on-budget. At the Department of Software Engineering at the Rochester Institute of Technology, we developed a course called Engineering Secure Software, designed for applying security principles to each stage of the software development lifecycle. As a part of this course, we developed a component called Vulnerability of the Day, which is a set of selected example software vulnerabilities. We selected these vulnerabilities to be simple, demonstrable, and relevant so that the vulnerability could be demonstrated in the first 10 minutes of each class session. For each vulnerability demonstration, we provide historical examples, realistic scenarios, and mitigations. With student reaction being overwhelmingly positive, we have created an open source project for our Vulnerabilities of the Day, and have defined guiding principles for developing and contributing effective examples.
 @article{MeneelyICSESEE2013,
  author = {Meneely, Andrew and Lucidi, Samuel},
  title = {Vulnerability of the day: concrete demonstrations for software engineering undergraduates},
  booktitle = {Proceedings of the 2013 International Conference on Software Engineering},
  series = {ICSE '13},
  year = {2013},
  isbn = {978-1-4673-3076-3},
  location = {San Francisco, CA, USA},
  pages = {1154--1157},
  numpages = {4},
  url = {http://dl.acm.org/citation.cfm?id=2486788.2486948},
  acmid = {2486948},
  publisher = {IEEE Press},
  address = {Piscataway, NJ, USA},
  abstract = {Software security is a tough reality that affects the many facets of our modern, digital world. The pressure to produce secure software is felt particularly strongly by software engineers. Todays software engineering students will need to deal with software security in their profession. However, these students will also not be security experts, rather, they need to balance security concerns with the myriad of other draws of their attention, such as reliability, performance, and delivering the product on-time and on-budget. At the Department of Software Engineering at the Rochester Institute of Technology, we developed a course called Engineering Secure Software, designed for applying security principles to each stage of the software development lifecycle. As a part of this course, we developed a component called Vulnerability of the Day, which is a set of selected example software vulnerabilities. We selected these vulnerabilities to be simple, demonstrable, and relevant so that the vulnerability could be demonstrated in the first 10 minutes of each class session. For each vulnerability demonstration, we provide historical examples, realistic scenarios, and mitigations. With student reaction being overwhelmingly positive, we have created an open source project for our Vulnerabilities of the Day, and have defined guiding principles for developing and contributing effective examples. }
}
 

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper Validating software metrics: A spectrum of philosophies Andrew Meneely, Ben Smith & Laurie Williams ACM Trans. Software Engineering Methodologies (TOSEM) vol. 21, no. 4 24:1–24:28 Feb. 2013
Context: Researchers proposing a new metric have the burden of proof to demonstrate to the research community that the metric is acceptable in its intended use. This burden of proof is provided through the multi-faceted, scientific, and objective process of software metrics validation. Over the last 40 years, however, researchers have debated what constitutes a "valid" metric. Aim: The debate over what constitutes a valid metric centers on software metrics validation criteria. The objective of this paper is to guide researchers in making sound contributions to the field of software engineering metrics by providing a practical summary of the metrics validation criteria found in the academic literature. Method: We conducted a systematic literature review that began with 2,288 papers and ultimately focused on 20 papers. After extracting 47 unique validation criteria from these 20 papers, we performed a comparative analysis to explore the relationships amongst the criteria. Results: Our 47 validation criteria represent a diverse view of what constitutes a valid metric. We present an analysis of the criteria's categorization, relationships, advantages, and philosophical motivations behind the validation criteria. We then present a step-by-step process for selecting appropriate metrics validation criteria based on a metric's intended use. Conclusions: The diversity of motivations and philosophies behind the 47 validation criteria indicates that metrics validation is complex. Researchers proposing new metrics should consider the applicability of the validation criteria in terms of our categorization and analysis. Rather than arbitrarily choosing validation criteria for each metric, researchers should choose criteria that can confirm that the metric is appropriate for its intended use by inspecting the advantages that different criteria provide. We conclude that metrics validation criteria provide answers to questions that researchers have about the merits and limitations of a metric.
 @article{MeneelyTOSEM2013,
  author = {Meneely, Andrew and Smith, Ben and Williams, Laurie},
  title = {Validating software metrics: A spectrum of philosophies},
  journal = {ACM Trans. Software Engineering Methodologies (TOSEM)},
  issue_date = {November 2012},
  volume = {21},
  number = {4},
  month = {feb},
  year = {2013},
  issn = {1049-331X},
  pages = {24:1--24:28},
  articleno = {24},
  numpages = {28},
  url = {http://doi.acm.org/10.1145/2377656.2377661},
  doi = {10.1145/2377656.2377661},
  acmid = {2377661},
  publisher = {ACM},
  address = {New York, NY, USA},
  keywords = {Software metrics, systematic literature review, validation criterion},
  abstract = {Context: Researchers proposing a new metric have the burden of proof to demonstrate to the research community that the metric is acceptable in its intended use. This burden of proof is provided through the multi-faceted, scientific, and objective process of software metrics validation. Over the last 40 years, however, researchers have debated what constitutes a "valid" metric.
  Aim: The debate over what constitutes a valid metric centers on software metrics validation criteria. The objective of this paper is to guide researchers in making sound contributions to the field of software engineering metrics by providing a practical summary of the metrics validation criteria found in the academic literature.
  Method: We conducted a systematic literature review that began with 2,288 papers and ultimately focused on 20 papers. After extracting 47 unique validation criteria from these 20 papers, we performed a comparative analysis to explore the relationships amongst the criteria.
  Results: Our 47 validation criteria represent a diverse view of what constitutes a valid metric. We present an analysis of the criteria's categorization, relationships, advantages, and philosophical motivations behind the validation criteria. We then present a step-by-step process for selecting appropriate metrics validation criteria based on a metric's intended use.
  Conclusions: The diversity of motivations and philosophies behind the 47 validation criteria indicates that metrics validation is complex. Researchers proposing new metrics should consider the applicability of the validation criteria in terms of our categorization and analysis. Rather than arbitrarily choosing validation criteria for each metric, researchers should choose criteria that can confirm that the metric is appropriate for its intended use by inspecting the advantages that different criteria provide. We conclude that metrics validation criteria provide answers to questions that researchers have about the merits and limitations of a metric.}
}
 

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities Yonghee Shin, A. Meneely, L. Williams & J.A. Osborne IEEE Transactions on Software Engineering vol. 37, no. 6 772–787 2011
Security inspection and testing require experts in security who think like an attacker. Security experts need to know code locations on which to focus their testing and inspection efforts. Since vulnerabilities are rare occurrences, locating vulnerable code locations can be a challenging task. We investigated whether software metrics obtained from source code and development history are discriminative and predictive of vulnerable code locations. If so, security experts can use this prediction to prioritize security inspection and testing efforts. The metrics we investigated fall into three categories: complexity, code churn, and developer activity metrics. We performed two empirical case studies on large, widely used open-source projects: the Mozilla Firefox web browser and the Red Hat Enterprise Linux kernel. The results indicate that 24 of the 28 metrics collected are discriminative of vulnerabilities for both projects. The models using all three types of metrics together predicted over 80 percent of the known vulnerable files with less than 25 percent false positives for both projects. Compared to a random selection of files for inspection and testing, these models would have reduced the number of files and the number of lines of code to inspect or test by over 71 and 28 percent, respectively, for both projects.
 @article{ShinTSE2012,
  title = {Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities},
  volume = {37},
  issn = {0098-5589},
  doi = {10.1109/TSE.2010.81},
  abstract = {Security inspection and testing require experts in security who think like an attacker. Security experts need to know code locations on which to focus their testing and inspection efforts. Since vulnerabilities are rare occurrences, locating vulnerable code locations can be a challenging task. We investigated whether software metrics obtained from source code and development history are discriminative and predictive of vulnerable code locations. If so, security experts can use this prediction to prioritize security inspection and testing efforts. The metrics we investigated fall into three categories: complexity, code churn, and developer activity metrics. We performed two empirical case studies on large, widely used open-source projects: the Mozilla Firefox web browser and the Red Hat Enterprise Linux kernel. The results indicate that 24 of the 28 metrics collected are discriminative of vulnerabilities for both projects. The models using all three types of metrics together predicted over 80 percent of the known vulnerable files with less than 25 percent false positives for both projects. Compared to a random selection of files for inspection and testing, these models would have reduced the number of files and the number of lines of code to inspect or test by over 71 and 28 percent, respectively, for both projects.},
  number = {6},
  journal = {{IEEE} Transactions on Software Engineering},
  author = {Shin, Yonghee and Meneely, A. and Williams, L. and Osborne, {J.A.}},
  year = {2011},
  keywords = {Charge coupled devices, code churn, Complexity theory, developer activity metrics, Fault diagnosis, fault prediction, Linux, Mozilla Firefox Web browser, online front-ends, open-source projects, Predictive models, program testing, public domain software, Red Hat enterprise Linux kernel, security inspection, software fault tolerance, software metrics, software security, software vulnerabilities, source code, vulnerability prediction., vulnerable code locations},
  pages = {772--787}
}
 

A word cloud of the paper's text. Generated by TagCrowd.com