Case Study

Overview

The purpose of the case study project is to get you acquainted with the security challenges of a real, complex, messy software product. In class, you learn about security ideals, common mistakes, and best practices. In the case study, you will see how those ideals are applied (or not applied!) to actual problems.

This case study is designed to help you in two key ways: investigation and co-authorship. The investigative part of this project is to help you to learn about software projects from the outside in. This means reading bug reports, documentation, mailing lists, commit logs, and anything else you can get your hands on to understand what’s going on. The co-authorship part of this project is to help you learn what it’s like to describe complex arguments (i.e. specific security risks) to a technical, yet non-security-minded audience. The “co-” part is to help you learn how to write much like how you’ve learned how to code… communicating, coordinating, know when to work alone, know when to work collaboratively, giving good feedback, and reacting to feedback.

The main component of this project is a paper that will be written much like we write code - iteratively. Every few weeks we will have another chapter of the paper due, along with the revisions from the previous chapter. A significant grading emphasis will be based on how well you react to the instructor’s feedback, and how well you give feedback to others.

In terms of writing style, here’s what the instructor likes to see:

  • For formal, technical writing, brevity and clarity are key. I would far rather see one clear, thoughtful sentence than three repetitive sentences. You might notice that we have no word minimums or maximums here - only questions that must be answered.
  • Furthermore, the paper must have a cohesive writing style. Like coding, this is not something you want to be slapping together from a few paragraphs people wrote separately. The varying sections need to be all part of a single narrative. Thus, please do NOT repeat the questions below in your paper - it should read smoothly on its own. Also, using bullet points as a structure is not a cohesive style - this is not a PowerPoint presentation, it is prose.

Submissions and Deadlines

  • All of the work on the case study will be in the RIT GoogleDocs system, thus there will be no submissions to dropboxes in myCourses for this portion of the class. After the proposal, write your case study in the shared Google Drive folder for your section:

  • Deadlines are posted on the course schedule.

  • Note that the instructor can view the revision history on all GoogleDocs in case a deadline was in question.

  • All members names must be stated on the submission

Example Paper

This is an example paper on Chromium that was considered excellent work (PDF). This might give you an idea for the depth of research and quality of writing we consider to be great.

Formatting

Please use following formatting throughout your paper:

  • 10pt font
  • Justified text (no ragged edges).
  • Headers have a maximum size of 16pt font.
  • Tables & Figures must be numbered starting at 1 going through the whole document.
  • Tables must have their caption be above the table.
  • Figures have their caption below the figure.
  • Cite URLs by using hyperlinks in the text, like this. Unlike what you may have been taught in the past, you may cite internet sources in this paper.

Plagiarism

Stealing someone else’s writing is a despicable act that will not be tolerated. Do not copy text from other sources. For example, do not simply copy the project website’s description into your description

Case Study Proposal

You must find your own team for the project. (2-3 members per team).
Choosing a project may take some effort, as not every project out there makes a good candidate. Here are the minimum requirements for a good case study:

  • Must have a domain that has significant security risks.
  • Non-trivial. Large enough that one single person cannot know everything. Minimum of 10k lines of code or 20 developers. You will not need get this system to build.
  • Not too massive, either. Entire operating systems are not recommended, for example.
  • Must be actually used in production.
  • Source code must be available.
  • Must have had a list of reported vulnerabilities. The more detailed the records, the better (e.g. do they trace to bugs and source code patches? do they have defect tracking of their vulnerabilities?). Note: third-party sources such as the CVE or CVEdetails.com are NOT enough (and sometimes wrong). The project themselves need to acknowledge vulnerabilities and you must look into the project history yourself.
  • Your choice must not be the same as others in the class.

Your proposal must be written in paragraph form and it must provide a convincing narrative about how you will be able study your project empirically. In your proposal, include the following in any organization you like, but include the following:

  • The names of each team member.
  • Project overview:
    • What is the product used for?
    • What is the development team like?
    • Are there any corporate sponsorships or connections? Or is it primarily volunteer-driven?
    • How often does the project release?
  • Security risks:
    • What are some example security risks of this system?
    • What kinds of data does this product protect?
    • What are the ramifications if this software is compromised?
  • Provides links (using just a regular hyperlink) to the project website, the source code repository, issue tracker, past vulnerabilities, and any other relevant information.

For this proposal, create a document called “X Case Study Proposal” where X is the name of your case study project, and submit to myCourses.

Grading Scheme (10pts)

  • 6pts: Submit a case study proposal naming a project.
  • 1pt: Project matches the minimum criteria.
  • 3pts: Proposal demonstrates a surface-level understanding of the project’s security risks.

Chapter 1: Domain and Historical Analysis

The purpose of this chapter is to provide an overall assessment of the security risks posed by the domain of your case study. Your chapter must include:

  • Product Overview: A description of the product itself and an overview of the findings of this chapter. Discuss users, purpose, business objectives, development team, process, and any other information that might be relevant to understanding both the project and product. This section must be more in-depth than your case study proposal.
  • Product Assets: Provide a listing (i.e. a table or figure) and discussion of the product resources, functionality, and intangible properties that an attacker would want to exploit. Be as comprehensive as possible, considering the system’s environment. Here are some helpful questions to ask yourself about your product to build up your list.
    • What are the pieces of data that this system is supposed to protect? Be specific - use names of files, use case-study specific names. “User data” is too broad.
    • Where are the places that unexpected code execution could occur? Think both within your product and your product as a vector into a host system.
    • Is the overall availability of your product an asset to protect? Note that some products don’t really have availability as an asset, others obviously do.
    • What are the “things” that this system is supposed to protect?
  • Example Attacks: Provide some examples scenarios of how someone could attack this system. Highlight what elements of CIA could be compromised. Here are some helpful tips on this section:
    • Think about malicious actors for this system and what they would want to do to it and with it.
    • Think about your assets. Is there an asset compromised in your example attack that is not in your asset list? Or, vice versa, what would an attack one of your assets look like?
    • Go more specific than a misuse or abuse case - include both the “how” and the “what”.
  • Vulnerability History: Vulnerabilities are engineering failures - let’s understand them better at a qualitative level. Each member of your team must research one vulnerability from your case study and provide an analysis. For each vulnerability, you must provide the following:
    • A unique name, such as the CVE number.
    • A link to the development team’s acknowledgment of this vulnerability. NOT the CVE entry, but to the original team acknowledging that they fixed this vulnerability.
    • Link to the source code commit(s) (e.g. Git, SVN) where the vulnerability was fixed.
    • Link to the source code commit(s) where the vulnerable code was originally introduced. (Use git blame and other annotation tools to traverse the history of the code). This is going to be an informed judgment call on your part - people don’t usually track this data. Make this a part of the discussion.
    • Note that we are not asking for when a vulnerability was reported (typically these dates are just a few days prior to the fix and are uninteresting). From an historical perspective, I don’t care about how long it took them to fix the vulnerability, only how we can further prevent vulnerabilities like this one to happen in the future.
    • Additionally, discuss the vulnerability as an engineering mistake, covering the following questions:
      • In your words, what does the vulnerability allow?
      • From a coding perspective, what was the specific mistake that was made?
      • What software engineering practices could have helped find this problem?
      • What category of CIA does this fall into?
      • How long was the vulnerability in the system from the time it was introduced to fixed?
      • How many different developers worked on this source code file(s) from when it was introduced to fixed? (study git log and similar commands depending on their version control system).

Remember: you’re now a software archeologist. This is a research project as much as it is a writing project. Plan on taking a lot of time tracking down source control commits, release histories, bug reports, mailing lists, or whatever software development artifacts you need to track down. Feel free to also share your sleuthing strategies with your team and your class - there are many clever ways of mining software repositories for this information.

A few notes:

  • Create a separate GoogleDoc called “Case Study”. All subsequent chapters will be in this document.
  • You may recycle sentences from your proposal, if you like (e.g overview stuff). But expand upon it.
  • Use your own words. You may quote product’s website, but be sure to quote it - otherwise it’s plagiarism (see above).
  • Remember to keep the writing cohesive. I recommend setting a team writing deadline ahead of time, and giving a few days to revise. Even though the research is individual, the team should be collaborating heavily on the writing.
  • These research papers (1, 2) might be helpful in understanding how you can apply git blame to identify the introduction commit (although you won’t be automating it, you can apply it manually).
  • Cited, specific examples are better than generalities. Specificity is the soul of narrative.

Grading Scheme (60pts)

  • 5pts: Overview section.
  • 10pts: Assets section.
  • 10pts: Example attacks section.
  • 25pts: Vulnerability History section.
  • 5pts: Overall quality of the research.
  • 5pts: Reaction to any comments from the proposal.

Peer Review of Chapter 1

After your chapter is due, you will be assigned another group in the class to review. (Zhang Groups, Meyers Groups). You will review the chapter, and submit comments. You will graded on the quality of comments you provide, so please put thought into them.

Each member of your team will be individually responsible for submitting feedback. Place your comments in appropriate places in Google Docs, using your own login so that the comment can be attributed to you. Be available with clarifications on your comments if needed.

See the syllabus for exact due dates, but this will generally be due one week after the chapter 1 due date.

Your comments should fall into the following categories:

  • Grammar, Spelling, or Style: These comments should be trivially correctable. Don’t just look for low-hanging fruit here - think about what would make the writing smoother and easier for you to read. For example, “Use active voice in this paragraph instead of passive voice.”
  • Metholology or Investigation: These types of comments concern the content itself (i.e. the assertions about security that the authors are making). For example, “I think this vulnerability could also be an arbitrary code execution concern - can you discuss this possibility too?".

A high-quality comment is insightful, actionable, and constructive. We do not place a number on how many comments you are to give - only on how helpful and insightful they are.

Grading Scheme PER INDIVIDUAL (15pts)

  • 5pts: Actionable style or grammar comments.
  • 10pts: Methodology or investigation comments.

Chapter 2: Design Analysis

The purpose of this chapter is to assess the high-level design of the system. Your chapter must include:

  • Architecture Overview: Describe the architecture of the system. Please answer the following questions:
    • What are the subsystems? What does each subsystem do?
    • Are there subsystems that are expressly security features? (e.g. encryption, authentication) Describe these in more depth.
    • Consider the cost of developer mistakes in terms of subsystems. For example “if a developer makes a mistake in this subsystem, what happens?”
    • Do some subsystems appear to be more susceptible to code-level vulnerabilities?
    • Do you see security built into system at this architectural level? (e.g. distrustful decomposition)
    • How might compromising one subsystem affect the security (integrity, reliability, etc) of the others or of the system as a whole?
    • How does the system’s interaction with external systems put it at risk?
    • Consider how the architecture might change over time.
  • Threat Model: Build a threat model diagram and conduct an analysis using the Microsoft tool. Write up your analysis:
    • What are the machine boundaries of the system? Does this vary from deployment to deployment?
    • What are the trust boundaries of the system? Are there any trust boundaries more important than others?
    • Please make this diagram legible - I would rather you show the riskiest parts of the threat model than enumerate every possibility. As a guide, threat models get pretty confusing after about 12 primitives per diagram. If you feel your diagram must be more complex, feel free to break it up into multiple diagrams.
  • Assets to Threat Model Tracing: Provide a mapping from your assets in Chapter 1 to the elements in your threat model. There should be a figure or table that maps primitives in your threat model to the assets listed in Chapter 1. Discuss this mapping:
    • Does the location in the architecture affect the p(exploit) of an asset? Why or why not?
    • Are there locations in the architecture where no assets reside? Did you miss any assets there?
    • In your opinion, are there locations in the architecture where too many assets reside?
    • Use your terminology from the domain chapter (remember, this is all one cohesive paper!).

A few notes:

  • Inferring the architecture for the system can be a real challenge if it’s not documented. Here are a few places you can look for hints on what the subsystems are:
    • Bugs. Any custom fields, or what is mentioned in discussions.
    • File system layout.
    • Javadoc or other documentation.
    • API documentation for third-party developers can imply the subsystem layout.
    • You can always ask them on your mailing list. If you do, be sure to have the whole group review that email before it goes out.
  • Resolve any comments from the domain analysis. Feel free to respond to those comments with clarifications, or just contact the instructor directly.

Reaction to prior comments: Also due with this chapter is revision to your writing based on all prior comments from the TA, professor, or peer group. We are looking for reasonable revisions. Hit “resolve” on the comments when they are fixed.

Grading Scheme (60pts)

  • 15pts: Overview and subsystems.
  • 20pts: Threat Model diagram and description.
  • 10pts: Mapping to Assets.
  • 15pts: Reaction to previous comments.

Peer Review of Chapter 2

Same instructions and grading scheme as Chapter 1. Remember, grading for peer review is per individual student.

Presentation

Prepare a lightning talk for your case study. You do not need to cover every chapter of your paper. Instead, choose the most interesting part(s) of your case study. Each team will have a total of 6-7 minutes (3-4 minutes to present and 3 minutes for questions). A set of slides is not required, but is recommended. You may have one person present, or multiple people. Outside of extenuating circumstances, all team members must be present to receive credit for the presentation. Concreate examples are strongly encouraged.

Grading Scheme (35pts)

  • 15pts: Demonstrating thorough research of the software project.
  • 10pts: Engaging choice of content from the case study.
  • 10pts: Engaging style.

Grading Summary (195pts)

  • 10pts: Proposal.
  • 60pts: Chapter 1.
  • 15pts: Chapter 1 peer review feedback. Per individual.
  • 60pts: Chapter 2.
  • 15pts: Chapter 2 peer review feedback. Per individual.
  • 35pts: Presentation.