Sentinel Prime

Senior Project Final Self-Assessment

Team: Sentinel Prime

Project: Amazon EC2 Cloud Demonstration Environment

Sponsor: Paychex

Product

3. Did the team prepare all the documentation artifacts requested by your faculty coach and sponsor? Were these documents carefully inspected prior to delivery? How would you assess the quality of the document artifacts?

The team has completed both the required SE department documentation and the documentation requested by Paychex. Such requested documentation includes: an ‘EC2 Lessons Learned’ doc containing roadblocks and solutions the team encountered while dealing with EC2, and a documentation describing the source code components.

These documents were divided among team members, but each section has been reviewed by at least two members. Most of the document has been reviewed by at least three team members. The quality of the document artifacts are on par with the quality of the entire project: amazingly superior to anything a 361 team would have done.

4. How well did the team elicit the requirements? What approaches were used to elicit the requirements? Were key requirements missed? What methodology was used to document and validate the project requirements?

Requirement elicitation was very important to the team, and was a contributing factor when decided to use Scrum as the process. Through the use of user stories, the team could easily make a list of tasks and features that were subsequently reviewed and passed by the sponsors. Post-iteration meetings included the acceptance of the user stories and marked a requirement as complete. User story review and re-prioritization was also done during the post-iteration meetings.

5. Did the team explore the entire design space before arriving at a final design? Have there been many errors found in the design? Was it necessary to make major changes to any part of the design? What were the reasons for the change?

The team did consider many different ways to approach the problem. After thinking about it, the team placed the problem in the context of how the sponsors would likely use it, and made the decisions accordingly. For example, it was known that the sponsors would likely put in their own user authentication system using LDAP, so the team decided that putting a large amount of effort into writing one would not be beneficial.

Other design decisions were made through discussion within the team, and with external experts more experienced with working with applications on the cloud.

6. How has the development and implementation progressed? What percentage of the product do you estimate was completed? Is the team providing the documentation within the implementation artifacts?

The development and implementation progressed more or less on schedule. There were some features that did not get completed as scheduled between the 5th and 6th iterations, however planning was done so that very easy, low priority features would be worked on in the last iteration, inserting a good cushion for the overflow. Currently, the project is complete. Yes, the team is providing documentation with the implementation artifacts.

7. What was the team’s testing strategy? Did the team develop a test plan? If so, was it followed? Did the team performing unit testing? Did the team use any test frameworks, such as JUnit? What are the testing results? Were any major defects found during system test? If so, were they fixed? Did the team do regression testing?

The team was very dedicated to test-driven-development. WatiN was used for user-interface acceptance testing, as well as a standard unit testing suite with MBUnit. These provided a good way to do regression testing as well. The tests were most helpful in integrating new tasks, and figuring out how they would interact with existing components. Because of this, most of the quirky issues were found and ironed out before full integration. No serious defects were found.

8. Products need to be designed within guidelines and constraints appropriate for each project. It is also important to consider the impacts of the products that are designed. In the following categories discuss the constraints and impacts that have a bearing on your project. Note that all of these categories may not have bearing on your project but your project is probably affected by many of them.

• Economic issues

i. The use of the cloud will reduce Paychex’s wasteful spending by having a need-based balancing system. This will free up their corporate resources for a variety of things, one of which may be increasing the paychecks of their employees, introducing more liquidity into the market, and strengthening the economy.

• Environmental issues

i. Through the use of the cloud, Paychex does not have to set up servers, reducing energy usage.
• Social issues

i. By having a standard way of setting up demonstrations, Paychex sales reps can now assure equality in demonstration quality towards potential customers. Personal bias is significantly removed.

• Political issues

i. The R&D team has successfully created a proof-of-concept application with cost-saving implications for the company at miniscule cost by using an RIT Senior Project team as a free labor resource. They will likely have good political positioning within the company.

• Ethical issues

i. Paychex sales representatives will now feel empowered with the ability to set up their own demonstrations, largely independently. Empowered sales reps good deeds make.

• Health and safety

i. By reducing load on their datacenters by using the cloud, Paychex has reduced the changes that an IT admin will be electrocuted.

• Manufacturability

i. By paying Amazon to use their service, Paychex has increased Amazon’s ability to manufacture some of their other products, such as the Kindle.

• Sustainability

i. The cloud reduces Paychex dependency on their existing datacenters, moving towards a greener use of technology.

9. What industry and engineering standards was your project required to adhere to? Were these new standards that the team had to learn? Did your sponsor provide you support for understanding these standards? Did you have to educate your sponsor about these standards?

Paychex stipulated no standards on the project. This was a proof of concept project that was more about ‘working’ than ‘adhering.’

Process

1. What was your process methodology? Was the process appropriate for the project? Did you follow the process or modify it as the project progressed? If you could repeat the project, what would you do differently?

The team chose to use Scrum, an agile, iterative process. This was appropriate for the project because of the short timeframe of the project and the amount of interaction the team was able to have with the project sponsors. Given the time constraints, the team was able to complete 7 two-week iterations along with a lengthy iteration zero, and managed to meet with Paychex at the completion of nearly every sprint. Since the team was unable to meet every day to work on the project, stand up meetings were not part of the process. We kept a backlog and planned sprints based on story points. We changed the number of points per iteration as the project progressed, but no other major changes needed to be made to the process. Given the great success of the project, Team Sentinel Prime would definitely use the same process again.

2. Was there a large requirement to learn the problem domain? What approach was used to gain domain expertise? Did your sponsor provide adequately support? What forms of support did you receive?

Learning the Amazon EC2 domain was a challenge for the team. Since Paychex gave us this project so we could learn about this domain for them, and report back, we were given little guidance at first and there was a lot of trial-and-error learning as a result for the first few iterations and all of iteration zero. The team found tutorials online and used the technology to set up instances before beginning the project. We received advice and support from Paychex, but at times found little to no help for how to solve some of the problems we ran into with using the new technology. The team kept writeboards on Basecamp to share what we have learned with each other, so those resources were important since we could share what would directly affect the project.

3. What mechanisms did the team using to track project progress? Did they give the team and sponsor adequate insight into project progress and issues? How well did the team track its project progress? How often did these artifacts get updated on the department project website?

The team used two online tools, Basecamp and Pivotal Tracker, to manage the project. Pivotal tracker was used to keep track of the progress of the user stories, track bugs, and assign responsibilities to each team member. Stories were presented to Paychex when we felt they were finished, and they would eventually sign off on the stories as delivered. The team used this instead of email for communications, so there would be a record of every communication that was made, and anyone could see what was going on so they didn't feel out of the loop. The project sponsors had access to Basecamp and used it to communicate with us. Meeting minutes and meeting plans were made through Basecamp.

4. Did the team conduct effective meetings?

The meetings the team held with Paychex were all planned beforehand, so we knew what we needed to accomplish at the meeting, and had a record of what questions we needed to be answered. Other team members usually consisted of each of us working alone or in pairs on the tasks for that iteration. In general, the meetings were effective and attended by all members.

5. Did the team meet all project milestones? Which milestones, if any, were missed or were met ahead of schedule? What contributed to schedule changes? What could the team have done differently to ensure that milestones were met?

Team Sentinel Prime met all milestones. Some requirements were changed through the course of the project, but the team finished all work assigned to them by Paychex. The team had to adjust the schedule a bit so all the requirements could be met, notably extending requirements from the iteration with database work into the final iteration, but this was done successfully.

6. Was the team required to adopt new technologies? What were these technologies? What approach did the team use for selecting the appropriate technology for the project? Did the sponsor provide any support for learning these technologies? How well did the team ramp up on the new technologies and begin to apply them effectively?
The team had to learn ASP.Net MVC2 and the Amazon AWS SDK for the project. The team found tutorials on both technologies to follow to get up to speed. The team chose to use ASP.Net MVC2 because of all the groundwork it provided to get started initially, and the structure of the code provided was easy to use. The sponsors understood the team needed some time to get up to speed with the new technologies, and allowed us time to have an iteration zero where we were not providing them with functionality for any of their requirements, but just getting up to speed. The team was awesome at learning the new technologies and ready to code when we started iteration one.

7. How well did the team maintain quality control over the project artifacts? Have all artifacts been reviewed for adherence to quality standards? What was the review process used by the team?

The team was serious about testing, and had a test suite that would run each time something was checked in. All of the artifacts have gone under the same testing, and we have made and recorded bug fixes in Pivotal Tracker. The team would make sure the build was passing and also manually tested everything to ensure it worked the way each team member expected.

8. Did the team have any issues with configuration management? How were these problems solved? What percentage of project artifacts is under configuration control?

The team had no issues with configuration management. All of our artifacts have been under configuration control (using the SVN servers provided by SSE) since the beginning of the project.

9. What was the set of metrics that the team tracked? Did the team gather these metrics on a consistent basis? What did the team learn from the review of these metrics?

Metrics included code complexity, code coverage, feature burndown and team velocity. The team tracked these every two weeks, corresponding with the end of each iteration. The code complexity metric added little value, as the ASP.NET MVC 2 framework added a significant amount of complexity to the code out-of-the-box. The code coverage was used as a way to make sure the testing was sufficient. Feature burndown was very useful in eyeballing progress, and team velocity was a key factor in the planning of future iterations.

Communication and Interaction

1. How well did the team communicate project progress to the sponsor? What regular communication did the team have with the sponsor? Did the team been maintain this communication to the satisfaction of the sponsor? Were any adjustments needed in the communication over time? Were these changes initiated by the team or the sponsor?

At the very least, the team communicated progress to the sponsor at the end of every iteration. However, there were times when the sponsor asked a question mid-iteration, or the team had a question for the sponsor. In such cases, the team used Basecamp as a means of communication. This was very effective. The sponsor has not expressed disappointment with the level of communication.

2. Did the team need to provide technical input to the sponsor? How well did the team educate the customer in these areas? What mechanism did the team use?

At the end-of-iteration meetings, the team discussed work completed, including a technical overview of the approaches. The sponsors were software folk, so they understood nearly all concepts. If not, they had a contact within the company that helped discuss some of the problem, such as the case with Citrix.

3. Was this an effective team? What has been contributing to and detracting from the team’s effectiveness? What are the team’s weak points? What are the team’s strong points? What changes could the team have made to make it more effective?

This was an effective team. Contributing factors include rapid development cycles with fast feedback, strong technical skills among the members, and specialization of the team members across the various parts of the project. Also post-mortems were occasionally performed when particular weaknesses were seen and fixed so that the harm was isolated to as small amount of time as possible. The only controllable change was to have the post-mortems rigorously after each iteration.

4. What mechanism did the team use to communicate with the faculty coach? Was communication with the coach effective? Were there any trouble spots with the faculty coach communications? What could the team or faculty coach have changed to make their communication more effective?

Communication with the faculty coach was done mostly in person, although email was also used. Basecamp was used occasionally. There were no trouble spots.

5. Did the team need to interact with department staff personnel, i.e., the office staff or system administration? Was this been handled in a professional manner? Were there any problems with these interactions?

The team interacted with the SE department personnel to open up the conference room. This was done in a very professional manner, and no problems were encountered.

6. Does the team have a complete website with all project artifacts stored and up-to-date on the software engineering department webserver? How often were entries on the webserver updated?

The team had a complete website, although artifacts were not all stored on it, and were often out of date. However, all artifacts were kept up to date in the repository, which the team used liberally. Neither the sponsor or team coach expressed a need in up-to-date artifacts on the website, so the effort was put elsewhere. Team times and metrics were typically reported at the end of each iteration, once every two weeks.

7. How well has the team made presentations to the sponsor and faculty coach? Was the final project presentation done in a professional manner? Was the poster presentation done in a professional manner? What could have been done to improve the team’s presentations?

We definitely feel we put forth a lot of effort into both presentations, making them accessible to those outside of our problem domain and still providing a great deal of technical details. Like any presentation, more practice could have helped, but we did well.

8. Does the technical report adequately document the project and its results? Was the paper of high technical and editorial (language, style, grammar, etc.) quality? Did all teammates contribute to the paper? Did the sponsor contribute to the paper? Did the sponsor review the paper?

The report will seek to cover as much of the project as we possibly can. The sponsor did not contribute or review to the paper, and was concerned more with getting a feedback document on lessons learned when using Amazon EC2.

9. How well did the team work with other senior project teams, coordinating access to lab space and equipment, sharing experiences and ideas, etc.?

The team did not share any resources with other teams, and used the lab rooms whenever possible to hold meetings.

Achieving Customer Satisfaction

1. In the team’s opinion did the work satisfy the project sponsor? Were there any weak spots in this regard?

The sponsor has expressed approval with the team’s work. All initial features were implemented, and the team has been very open to suggestions regarding the ease of turning the project over. The only weak spot would have been to formally push for satisfaction through the use of a survey or questionnaire.

Achieving Team Satisfaction

1. Did the project satisfy the team’s expectations for learning? Were there any weak spots in this regard? What could have been done differently to improve the team’s learning experience?

Team Sentinel Prime has learned a lot about EC2, ASP.NET MVC, automation strategies and Scrum. Team members openly shared information to other members to increase learning without obvious judgment. The learning experience has been grand.