We will use the Qualtrics survey tool (http://survey.rit.edu which redirects to https://www.rit.edu/survey/) for doing peer evaluation twice during the term. This is the same tool that we used for the Personality Types and Team Formation surveys.
The SWEN-261 peer evaluation template has three questions: rate yourself and each team member on a 5 level scale, describe the rating for each team member giving kudos for the positive and identify areas for improvement, provide confidential feedback to the instructor. The first two questions are required. The wording for the first two questions indicate that the instructor may provide information about the responses to the team member. The third provides a way for the team members to provide confidential information just to the instructor.
You can define a survey by importing either a text file or a .qsf file which is a Qualtrics defined file format using JSON formatting within the file. The .qsf file captures all of the options for a survey. You can also export an existing survey in the Qualtrics format.
Here is a suggested workflow for doing your peer evaluations.
Several peer evaluation survey templates were defined and exported. You can use one of these to define your peer evaluation survey. These template files have placeholder tags for elements that will change from section to section.
The template files are below. You will need to select the one making sure that it .
Team Peer Evaluation Template - multi 4-5s.qsf - four teams of fiveTeam Peer Evaluation Template - multi 5-4s.qsf - five teams of fourTeam Peer Evaluation Template - multi 5-6s.qsf - five teams of sixTeam Peer Evaluation Template - multi-team.qsf - the original file just in case. This should be the same as five teams of sixTeam Peer Evaluation Template - single 4-person.qsf - single four person teamTeam Peer Evaluation Template - single 5-person.qsf - single five person teamTeam Peer Evaluation Template - single 6-person.qsf - single six person teamTeam Peer Evaluation Template - single team.qsf - the original file just in case. This should be the same as the single six person teamStarting with of the survey template covers the number of teams you have and the largest team that you have, you will create the Qualtrics project by importing a .qsf file. If you import the template .qsf file then you will need to manually edit the placeholders to represent the data for your section. Alternatively, you can provide a tabbed teams file that specifies which team each student is on and use that with one of the template files as input for the make-peerevals-surveys script which will generate the .qsf files with placeholders replaced with the information for your section.
make-peereval-surveys is a Perl script that takes as input a tabbed teams file and a survey template .qsf file. The script only uses modules that are in a standard Perl install. The script scans the .qsf file and replaces the placeholder tags with the specific information for your section. There are more placeholders than you will most likely need. When you do the next step of importing the .qsf file to create a survey project, you will need to delete the placeholders that you are not using.
The tabbed teams file holds one student per line with each line being team-id<tab>student-name<tab>rit-login-id.
Here are the two typical command lines that you would use to generate .qsf files for your section:
perl make-peereval-surveys -t "SWEN-261 Section ss Mid-Term Peer Evaluations" tabbed-teams.txt "one-of-the-multi-team-files.qsf"ss is your section number. This will generate a single file, Peer Evaluations.qsf,
which you will create a single Qualtrics project for all the teams to complete by importing this .qsf file.perl make-peereval-surveys -s -t "SWEN-261 Section ss Mid-Term Peer Evaluations" tabbed-teams.txt "one-of-the-single-team-files.qsf"ss is your section number. This will generate one file for each team, Peer Evaluations-t.qsf,
where t is the team id for the team. You will create a separate Qualtrics project for each team by importing that team's .qsf file.
The script generates a number of messages to STDERR for error conditions. If your template file is not an exact match for the composition of your teams, you will most likely encounter messages indicating that information for a particular team or team member was not provided. Look carefully at messages at the end when the script checks if teams or team members that were defined never had any of their information output. This usually results from entry errors in the team information file.
Here is the -h help information for make-peereval-surveys which provides information on the format of the tabbed teams file and the placeholders.
Usage: make-peereval-surveys [-hsv] [-b basename] [-n teamname-file]
[-t survey-title]
tabbed-team-info-file [survey-template-file]
-b provide filename prefix for all generated surveys
-h output this message
-n provide long form names for teams
-s separate survey file for each team
-t specify a title for the survey [[SURVEYTITLE]]
-v generate verbose output
The tabbed-team-info-file is a tab separated file with one line per student
holding team-idstudent-namerit-login-id. The team-id is a unique
id for each team, typically, the letter designating the team. Teams are
ordered by team-id. The student-name is the full name for the student. Team
members are ordered by full name. If a comma exists in the full name, it is
assumed to divide the full name into last-name, first-name which would be
available individually in substitutions. The script generates a message if it
finds that the template file uses last-name or first-name, and some names were
not provided using the comma notation. Blanks would be output for those
students instead of their names. The student-name can be enclosed in quotes
which are stripped off. The rit-login-id is just the id part, i.e. it does not
include \@rit.edu.
The tabbed-team-info-file is required. The survey-template-file can be
given on the command line or read from standard input.
The teamname-file is a tab separated file with one line per team
holding team-idlong-form-team-name.
The survey template file is gotten by exporting a Qualtrics survey that
defines the template in their QSF format and then pretty printing it to
a file. https://jsoneditoronline.org is a good website to use for this.
The script will copy the survey template to the output file substituting
for the keywords found in the template. The output filename for a multi-team
survey is filename-prefix.qsf. For single team surveys, the survey for each
team is placed in a file named filename-prefix-id.qsf where id is the team-id
for the team represented by the file. The default filename-prefix is
"Peer Evaluations" which can be changed using the -b option on the command
line.
The script will substitute the appropriate team and member specific values
for the following keywords.
[[SURVEYTITLE]] - a title for the survey set with the -t command line option
or using the default "Peer Evaluations".
[[TEAMNAME t]] - the name for team t where t is the team's numeric index in
the list of teams. If a long form name was given for the team that is used.
Otherwise, the team-id is substituted.
[[TEAMID t]] - the team-id for team t where t is the team's numeric index in
the list of teams.
[[TEAM t format]] - information about team t where t is the team's numeric
index in the list of teams. The format string can be any normal characters.
HTML constructs should be preserved. Within the format string substituions
are made for the following keywords TEAMNAME, TEAMID, and MEMBERS. The
MEMBERS keyword will generate a comma-separated list of all the members of
the specified team. MEMBERS can have an additional format specified in
parentheses, ie. MEMBERS(format). The format string can include the
keywords FULLNAME, LASTNAME, FIRSTNAME, and EMAIL. This represents the
format for outputting each team member's information.
[[MEMBERS t format]] - a comma-separated list of the members of team t where
t is the team's numeric index in the list of teams. Format specifies how to
output each team member's name and can include the keywords FULLNAME,
LASTNAME, FIRSTNAME, and EMAIL.
[[MEMBER t m format]] - information about team member m on team t where m is
the member's index in the list of members on the team, and t is the team's
numeric index in the list of teams. The format string can include the
keywords FULLNAME, LASTNAME, FIRSTNAME, and EMAIL.
When you have the .qsf file that defines the survey you want, follow these steps to import it to create a new project for the survey.
.qsf file that defines the survey..qsf filename as the project name.You most likely will need to remove some entries for teams and members who do not exist in your section.
When you have finished editing the survey, you will need to publish it. You can select the green Publish button on the righthand side toward the top of the survey and receive an anonymous link that you will distribute to the students to complete the peer evaluation. You can also select the Distributions tab toward the left side in the menu bar near the top of the page and then click Get a single reusable link to get the anonymous link. Any time that you do something to the survey later, you will need to publish it but the anonymous link does not change.
If you need to get the anonymous link later, select the Distributions tab, and then Anonymous Link to recover the link.
If you want to set a due date after which the survey closes, select Surveyfrom the menu bar, and then Survey Options from the second level navigation bar. In the middle of the Survey Options dialog box, select Survey Expiration and adjust the valid date range as appropriate.
Coming soon
The first thing that you will probably want to do is look at the submitted responses. You can look at the responses from the Data & Anlaysis or Reports navigation item.
The make-peereval-reports Perl script accepts the CSV output from a Qualtrics peer evaluation survey and can perform the following operations;
To use make-peereval-reports, you need to download the data from your peer evaluation survey project. Follow these steps to download a CSV file for your project.
If you are going to provide the feedback comments to the students, you will need to edit them to make them as anonymous as possible. You can do this is in the downloaded CSV file, or in the processed output generated by make-peereval-reports.
You will need to have Perl installed to run make-peereval-reports. The script uses one module (Text::CSV) that is typically not in the standard
Perl distribution. If you are working in the Windows cygwin environment, you can install Perl and Text::CSV as part of the Perl install. Look in the Perl category when running the cygwin setup program.
On other system with Perl installed then you should also
have cpan available which can do the installation for you. To install the module run: cpan Text::CSV. You will be asked to continue. You want to use local::lib for the
installation. The default answers for the questions asked
while the install is going on seem to work OK. There is a lot of stuff that happens
so sit back and get a cup of coffee. After the install is done, you may have
to log out and back in for it to work.
make-peereval-reports generates three output files (default names): evaluation data (PeerEvals.txt), confidential comments (PeerEvals-confidential.txt), ratings and factors (PeerEvals-ratings.csv). The information for individual students is identified in each of the output files. Command line options allow you to specify which output files the script should generate and to change the PeerEvals default filename prefix. The default evaluation output is simple text output of the data for each student. The script allows you to customize the look of the output by defining a boilerplate file that it will use when outputting the evaluation data.
You will use the same tabbed team file that you used when creatting the .qsf survey file to define your Qualtrics survey project. Assuming that you started with one of the provided survey template files, want to use the text boilerplate file, boilerplate.txt, for output, and have instructor comments in a file named instructor.txt, you would run make-peerevals-reports with the following command:
perl make-peereval-reports -ecr -p "Mid-Term peer evals" -i instructor.txt -k "FIRSTNAME LASTNAME" -b boilerplate.txt \
tabbed-teams.txt your-data-file.csv
Here is an example of the format for an instructor comment file.
[[Team a]] Your comment about team a which can be on multiple lines. [[Joe Smoe]] This would be comments about Joe Smoe which also could be on multiple lines. [[Mary Williams 1.00]] You have some comments on Mary Williams' work and you feel that you should override the factor that was computed. [[Team b]] Now your comments on team b. [[Sue Jones 1.50]] Sue got an override of the factor also. [[Team c]] General comments about team c but nothing for individual students.
Two sample boilerplate files are available below. As the help output below indicates, when the script detects one of the special keywords in the boilerplate file, it will substitute the appropriate information for the individual student. The output is formatted text. If the extension of the boilerplate file is .html, the script outputs the peer comments from the team members as an HTML <ul> structure.
You can use the output formatted using these boilerplates to provide peer evaluation feedback to the students. Browsers should be able to display the HTML output which you can then copy-and-paste into individual mail messages to the students, or place in a gradebook item such as Individual adjustment where you will enter the adjustment factor. The paste operation typically preserves the formatting.
The boilerplate files in parentheses are formatted to use the send-bulk-email script to email individual reports to each student. (See the caveats on the use of send-bulk-email below.) If you are going to use the direct email templates, you will need to edit/remove the header fields as appropriate for you. You also might want to add your standard signature at the bottom.
Here is the -h help output from the script which explains the
options that you have available to you and the substitution keywords identified in the boilerplate file.
Usage: make-peereval-reports [-hcrev] [-b boilerplate] [-i instructor-file] \
[-k FORMAT] [-p prefix] tabbed-team-info-file [raw-peerfile]
-b provide a boilerplate file to use for evaluation output
-c output confidential comments
-e output evaluation data
-h output this help message
-i instructor comments on team, individual, overriding factor
-k set the format for the key used to index students
-p provide filename prefix for all generated files (default: PeerEvals)
-r output ratings and factor data to csv file
-v generate verbose output
The tabbed-team-info-file is a required tab separated file with one line per
student holding team-id<tab>student-name<tab>rit-login-id. The team-id is a
unique id for each team, typically, the letter designating the team. The
student-name is the full name for the student. If a comma exists in the full
name, it is assumed to divide the full name into last-name, first-name which
would be available individually in substitutions. The script generates a
message if it finds that the boilerplate file uses last-name or first-name,
and some names were not provided using the comma notation. Blanks would be
output for those students instead of their names. The student-name can be
enclosed in quotes which are stripped off. The rit-login-id is just the id
part, i.e. it does not include @rit.edu.
The raw-peerfile is the CSV data downloaded from the Qualtrics peer evaluation
project or projects. It can be given on the command line or read from standard
input.
By default, the full name is used as the key to identify each student in the
survey data. The -k option can define the format for a different key to
use. The format can include the keywords: FULLNAME, LASTNAME, FIRSTNAME,
LOGINID.
Without a boilerplate file specified, the evaluation data is output in a
compressed text format. It includes full name, login id, factor, ratings, and
the team member feedback comments.
If a boilerplate file is specified, it is used as the base for the output of
each student's peer evaluation information. In the boilerplate file, you can
specify the following keywords that will be replaced with the corresponding
information for a student: FULLNAME, LASTNAME, FIRSTNAME, LOGINID, EMAIL,
FACTOR, RATINGS, PEERCOMMENTS, INSTRUCTORTEAM, and INSTRUCTORSTUDENT. If the
extension on the boilerplate file is .html, the PEERCOMMENTS are output as a
<ul> with one student comment per <li> element.
The instructor file allows the instructor to provide comments about a team's
performance, the performance of an individual student, and to override the
calculated adjustment factor for a student. Headers in the file identify the
comments given on the lines following the line with the header. Headers have
the following formats:
[[Team team-id]] team-id is the short id used to identify each team
[[student-key]] student-key is the key that is used to identify each student
[[student-key override-factor]] if a numeric value is given it will override
the computed factor for that student
You only need to provide headers for comments that you make. The header is on
a line by itself. All of the following lines of text up to the next header or
the end of the instructor file are associated with that team or student and
will be substituted for keywords in a boilerplate file. A default message of
"No instructor comments" is inserted when no comments are given.
"PeerEvals" is the default prefix for all output files. This can be
changed with the -p option. The default filenames are:
Evaluation data - PeerEvals.txt (.html if boilerplate extension is .html)
Ratings data - PeerEvals-ratings.csv
Confidential comments - PeerEvals-confidential.txt
The script detects a fair number of problems in the input data. It will identify students who did not provide any evaluations or are not known from the team file. It should continue running skipping over their data or evaluations. You may occasionally get "smart" quotes or other funky characters in the input data. The script handles these by translating them into innocuous equivalent characters.
Coming soon
WARNING: some of the material below is no longer correct or relevant
Here are the instructions for creating the peer evaluation and posting a link to it.