With Crowdmark’s Ready to grade workflow, you can scan and upload existing assessments in any format and grade them using Crowdmark’s online grading tools. Scan your completed assessments using the instructions here.

If you’d like to create an assessment template using tools such as anonymized grading, multiple choice auto-grading or automatic student matching, check out our Crowdmark booklet workflows.

Creating a Ready to grade assessment

1. To begin, click Create assessment. Name your assessment, choose Administered (In-person) and select the Ready to grade option. Review your selections and click Done. You’ll be taken to the Assessment setup page.
2. In the Template section, upload a blank version of your assessment. Crowdmark will use this template to recognize and separate the assessment booklets when you upload them. Verify the template looks correct using the preview provided, and click Save and continue to move to the Questions page.
   Note: because Ready to grade assessments do not require a dedicated cover page, all pages will be visible in the grading interface. Student anonymity is not guaranteed.
3. Define the questions on your assessment template using the instructions here. This is how Crowdmark will know where the questions are located for grading.

If a student claims that there was an issue completing or submitting an assessment, you can see a log of their submission activity. The log shows all the information Crowdmark has on the status of a student’s submission. Students can also see the log by following the instructions on the Verifying that an assignment was submitted page.

Viewing a student’s submission activity log

1. On your assessment page, click on Students in the sidebar.

   

2. You will see a list of students. Find the student you’re looking for and click on the three-dot menu that appears at the end of the row to access the student menu. Choose Activity log from the menu.

3. A panel will appear containing all the actions the student has taken within the assessment. To leave this view, simply click the close icon positioned in the top-left corner of your screen or click outside the panel.

What do the log entries mean?

Viewed assignment page

The student has accessed the assessment page where they can view assessment content and upload their responses. In a Timed assessment this action means a student viewed the gatekeeping page before starting their assessment, and has not viewed the assessment content yet.

Started Timed assessment

If this is a Timed assessment, the student has clicked through the gating page and viewed the assessment content. This is also the time that the assessment timer started to count down.

Student was added to a group/Student was removed from a group

If this is a group assessment, the student was added to or removed from this group’s assessment.

Uploaded x pages to question/Removed x pages from question

The student either added a response by uploading pages or removed previously uploaded pages. These responses are automatically saved in the student’s submission when added, but they’re not uploaded to Crowdmark’s grading system until the student clicks Submit.

Answered x question

The student typed an answer (Text response) or selected their answer (Multiple Choice). These responses are saved automatically in the student’s submission when they are added, but they are not uploaded to Crowdmark’s system for grading until the student clicks Submit.

Clicked submit button

The student pressed the Submit button to make their work available for grading.

Submission time logged

The Submit button click is registered by Crowdmark’s system. Crowdmark uses this time to calculate the student’s late penalty.

Submission processing complete

The student’s work is now uploaded, organized and ready for grading.

Upload failed

The student’s work failed to upload and an error message was shown.

Viewing file details

MD5 image hash

Crowdmark logs the MD5 hash (the file’s unique identifier) for each file that a student uploads to an assessment. What that means is if a student submits a file, then submits the same file again, the two hashes will be identical. The MD5 hash is built into the file when it is created on the student’s device or computer. If the student has altered the file by changing the format, editing the image, or even re-saving the same file, the hash will appear different from the original image hash.

If a student has an issue while submitting, or added files but never submitted their assessment prior to the due date, an instructor is able to upload and submit on their behalf. You can use the image hash to verify the image you are uploading is the same image the student uploaded prior to the assessment due date. There are 2 ways to verify the student file hashes:

1. Use a 3rd party MD5 hash verification site to check the hash on the file you have received from the student. Make sure the hashing method is set to MD5, and match the hash to the one in the student’s submission log.
2. Upload and submit the student’s file to Crowdmark, and verify the two hashes match in the Activity log. Keep in mind this action will overwrite the student’s previous submission.

Here’s an example of what MD5 hashes look like in a submission log. By default, the hashes won’t be visible. Click the Show file details button at the top of the log to show the hashes beside each file.

Troubleshooting a student’s submission attempt

Errors before submission (not recorded in Activity log):

• Incorrect file type: Crowdmark accepts PDF, JPG, and PNG files. The student must save or export their file in one of these formats.
• File size is too big: The limit is 12mb per image file (JPG, PNG) and 25mb per PDF file. The student should scan their work in lower quality.
• If the student tries to close the browser before submitting, they will see a popup alert asking them “Are you sure you want to leave this site? Changes you made may not be saved.”

Errors after clicking Submit (recorded in Activity log as upload failures):

• Sorry, your submission could not be uploaded: This is usually temporary and happens when something interfered with the upload. We recommend that you:

  • Try again later or use a different internet connection.
  • Disable extensions/add-ons in your browser or try a different browser.
  • Try a computer if you are submitting from a mobile device.

A student will know their assessment was not uploaded for grading if they do not see the “Thanks for your submission!” confirmation page. This troubleshooting is also covered in the documentation for students, linked from every submission page: Completing and submitting an assessment.

If these tips do not solve the problem, the student should contact Crowdmark Support and include the error message, the files they’re trying to submit, and the browser and operating system.

After the due date, students can’t edit or resubmit their work. Instructors and facilitators can edit and resubmit for students at any time (even after grades have been returned). To submit on behalf of a student or modify a submission, you can follow the instructions below to go to the student’s submission and enter responses or upload files as the student would.

Occasionally, a student may have completed a portion of their assessment, but encountered difficulty when attempting to submit (e.g., poor internet connectivity). If you navigate to their submission, you should see any text or multiple choice questions the student has completed successfully so far. If a student has uploaded files to PDF/image upload questions, you should be able to view the files in the student’s submission. However, if the student’s connection had issues with auto-saving, or if the student was working in multiple tabs or devices, some work might be missing. To resolve this issue, please ask the student to email or share with you any missing work. You can verify you have been sent the correct files using the MD5 hash present in the Submission log.

Any changes you make to the responses will be automatically saved and visible to the student if they view their submission. You must click the Submit button to make the student’s work available for grading.

If you submit past the due date, the student’s submission will still be considered late and any penalty you have set will be applied. You can adjust the student’s penalty by following the instructions on this page: Accommodation options for assigned assessments.

Note: If you only need to correct the rotation of uploaded pages, you do not need to resubmit the student’s work. You can do this while you are grading by following the instructions here: Rotating a page of student work

1. In your assessment, click on Students in the sidebar.

2. You will see a list of students. Find the student you’re looking for and click on the three-dot menu that appears at the end of the row to access the student menu. Choose Submission from the menu.

3. A full screen view of the student submission will appear. To leave this view, simply click the close icon positioned in the top-left corner of your screen or click outside of the panel.

4. Here, you can upload files, enter responses, or make changes to the submission if needed.

If the student has sent you files to upload on their behalf, drag and drop the student’s files to the upload areas under the questions or browse to locate them. You can drag pages between questions.

If you need to rotate a page, click the magnifying glass icon on the page, then click the Rotate page button. To remove a page, click the “x”.

5. When you’re ready, click Submit for evaluation.

Once the submission has completed, you will see a submission confirmation message.

Scroll through the submission below to ensure that the student’s work appears correctly.

A student may have lost, deleted, or not received the email containing the link to submit their assessment. You can see the status of your students’ emails anytime and resend them if needed.

Resending emails

1. In your assignment’s Dashboard, click on Students in the sidebar.

2. Find the student you’re looking for and click on the three-dot menu that appears at the end of the row to access the student menu. Choose Email history from the menu.

3. A panel will appear containing a list of all emails previously sent to the student. Here, you can review the email address used, the status of the last email sent, and troubleshoot any failures.

Click Resend email to attempt to send the same email again to a student. To leave this view, simply click the close icon positioned in the top-left corner of your screen or click outside the panel.

Understanding email statuses

Here are the most common statuses and what they mean:

• Sent: Email has been sent successfully.
• Opened: Email has been sent successfully and the student has opened the email.
• Failed: Something went wrong and the email failed to arrive. Try resending the email.
• Bounced: The email failed to arrive. Usually, this is because the email address doesn’t exist. It can also happen when a student’s school email account is attempting to forward to a personal email account but is incorrectly configured. Your institution’s IT department should be able to assist with this issue.

Why isn’t the “opened” status 100% accurate?

To detect when a student has opened an email, the student’s email client must be set to display images (some institutional email programs accept only plain text emails). If the status says the student has opened the email, the student has opened the email. However, if the status says the email is unopened, this could mean that the email was never opened or that the student’s email client simply isn’t displaying images.

To enroll students in your course in Crowdmark, you will need a list of student emails and, optionally, the students’ names, ID numbers and sections. Your student roster can be accessed from the Students page in your course.

Enrolling students

Students can be enrolled in two ways: by uploading a CSV file with your student roster or by pasting a list of student emails. An email address is all Crowdmark requires to enroll a student, but it is strongly recommended to upload students’ names and ID numbers as well. Some features, such as automated matching, need this information.

Students will not see the course or get any notifications from Crowdmark until you have set up an assessment.

Enrolling students with a CSV file

A CSV file can be exported from a spreadsheet using Excel or your preferred software. Your file should contain your full student roster with the following columns:

• Email
• Name (or separate First Name and Last Name)
  Do not include middle names.
• ID Number
• Section (if your course has sections)

An example CSV file:

To enroll students:

1. In your course, click Course Students in the sidebar.

   

2. Click the Enroll students button. A modal window will appear.

   

3. Click the Upload CSV file button. Locate the CSV file on your computer and upload it.

   

4. If the upload was successful you will see the number of students that were found in the file and be asked to map the CSV columns to Crowdmark’s data structure. Based on your column names, Crowdmark will guess the data that they contain. Make sure that each column is mapped correctly. Any columns that do not include student Emails, Names, Student ID Numbers or Sections should be ignored.

   

5. Once everything is mapped correctly, click the Continue button. You will see a list of students that will be enrolled for confirmation.

   

6. Click the Save button to confirm and enroll the students to your course.

Enrolling students by email only

If you cannot upload a CSV file, you can copy and paste a list of students’ emails to enroll them in the course.

1. In your course, click Students in the sidebar.

   

2. Click the Enroll students button in the upper right corner. A modal window will appear.

   

3. Click the Enter emails button. A text area will appear.

   

4. Paste or type your students’ emails and click the Enroll students button. The students will be enrolled in the course.

   

Updating, enrolling, and disenrolling students

You can update your students’ information, enroll, or disenroll students in a couple of ways: by uploading a CSV file with your student roster or manually in Crowdmark.

Updating, enrolling or disenrolling students with a CSV file

Crowdmark expects the CSV file you upload to include your full, current student roster. This means that any changes you make to student information in the CSV file will replace what is currently stored in Crowdmark. If you add a student to the CSV file, Crowdmark will enroll the student in the course. If you remove a student from the CSV file, Crowdmark will disenroll the student from the course.

When a student is enrolled in the course late, they will be automatically included in any assigned assessments where the due date has not yet passed (as long as “All enrolled students” or the student’s section is included in the assessment).

To update students:

1. In your course, click Students in the sidebar.

   

2. Click the Update students button in the upper right corner. A modal window will appear.

3. Click the Upload CSV file button. Locate the CSV file on your computer and upload it.

4. If the upload was successful you will see the number of students that were found in the file. If the file has different column names you may be asked to map the CSV columns. If not, you will see a list of students that will be updated and/or disenrolled for confirmation. If the student data columns are no longer mapped correctly, click the Change data mapping link to correct them.

5. Click the Save button to confirm and update the students to your course.

Updating a student manually

You can change an individual student’s information by locating them in the student list and selecting the three-dot menu that appears at the end of each row to access the student menu. Once selected, choose Profile from the menu to access the selected student’s profile.

The profile page displays personal info and submission status for all assessments the student is part of in the course. Click the Edit student button to modify name, student ID #, or section.

Disenrolling a student manually

You can remove a student from future assessments by disenrolling them on the Course students page. Keep in mind that if you upload a CSV file containing the student after manually disenrolling them, they will be re-enrolled.

1. In your course, click Students in the sidebar. You will see a list of students.

   

2. Find the student you’re looking for in the student list and click on the three-dot menu that appears at the end of the row to access the student menu.

3. Choose Disenroll from the menu to remove the student.

Sorting the student list

The student list can be sorted alphabetically (by first name), by email address, student ID or section. When you import your student list it will be sorted alphabetically A-Z by default. Use the arrows in your column headers to sort the student list as needed.

When you email your students the links to their assessment results, each student will be able to see images of his/her own assessment, with scores, feedback, and any other information you’ve chosen to include. Students will not see the cover pages of administered assessments as they contain personally identifying information.

If you change a student’s grades or feedback after results have been sent, for example if a re-grade is requested or work is added, the changes will happen in realtime on the student’s score page.

How can I see a student’s score page?

1. In your assessment’s Dashboard, click on Students in the sidebar.

   

2. You will see a list of students. Locate the student, then under the three dots menu on the right hand side, click Score.

   

3. A page with the student’s scores and feedback will appear.

4. If you have added an assessment summary message and/or chosen to include the Distribution of scores graph, the score page will have a Summary section at the top.

   

Accepting an invitation to proctor

You will receive an email from the instructor or exam coordinator inviting you to proctor the exam session.

1. Click Accept invitation in the email.

2. If you do not have a Crowdmark account, you will be directed to the Sign up page on Crowdmark. After completing the sign up process, you will be directed to the session page.

   If you already have a Crowdmark account, you will be prompted to sign in. After signing in, you will be directed to the session page.

You can now access Crowdmark and the session anytime by signing in to https://app.crowdmark.com/sign-in. After signing in, you can access the session again by clicking the Session ID in the My Sessions table.

The session page

After you have signed in, you will see the session page with the following sections.

• A Welcome section containing an introduction and any notes and instructions that the instructor or coordinator has provided.
• A sidebar on the right containing:
  
  • Session details: Date, time, location, and number of students. Click on the rolodex icon to see the list of students and their information.
  • Status: Click on the status to see your past activity with timestamps.
  • Contact info: For any questions you may have about the process (if available).
• Steps to administer the exam: Download and print booklets, Administer the examination, and Scan and save the uploaded booklets to your computer.

Administering the exam

1. Download and print booklets

You can download booklets for all students at once (a single PDF file with all of the booklet pages) or download each student’s booklet separately (a single booklet per PDF file). Click the Download button to save the booklet(s) to your computer. Print the booklets before the exam session will take place.

2. Administer the exam

Hand the booklets out to the students. Once the exam is over, collect the completed booklets.

3. Scan and save the uploaded booklets to your computer.

Remove staples from the exam booklets, ensuring that the QR codes in the upper corners are not cut off. For best practices and suggested scanner settings, see Scanning assessments. Ensure that pages are scanned as PDF or JPG files.

After the pages are scanned and stored on your computer, click Upload completed booklets. Select your scanned file(s) to upload. You will see the Upload history table fill up with pages. In the upper right corner you will see the number of booklets uploaded so far. Click this number to see which pages have not yet been uploaded.

If any “Undetectable QR” errors appear, click the Fix button to manually associate the page with the QR code information (the UUID) written in the top corner of the page.

Note: The front page of each booklet with student information (without the QR code) is for your information only and does not need to be uploaded. If the page is uploaded, you will see the error “Undetectable QR” until all other pages are uploaded. The error can be ignored.

How to locate the UUID:

Instructors have the ability to lock and unlock an assessment at any time. Locking an assessment makes it read-only, ensuring that scores and feedback cannot be changed once grading and regrading are complete. Assessments must be unlocked for grading.

Locking an assessment does not remove students’ access to work that has been graded and returned. However, it does block students from submitting, as well as prevent students from accessing their submissions before the grading process starts.

Locking an assessment

1. Navigate to your assessment’s settings page in the lefthand sidebar.

2. Select the Lock assessment button.

Once you’ve finished grading, you can download a list of grades. The downloaded CSV file can be opened as a spreadsheet and edited in a program like Excel or Google Sheets, or imported to your LMS’ gradebook. The grades CSV includes columns for:

• Links to students’ assessment results
• Name
• Email address
• Student ID
• Grades by question
• Question version (Randomized question groups only)
• Submission time (Assigned assessment only)
• Lateness penalties (Assigned assessment only)
• Submission status (Assigned assessment only)

You can also export grades for all assessments in a course at once. See Exporting a list of course grades for more information.

Exporting assessment grades

1. Navigate to your assessment and click Results in the sidebar.

   

2. Click the Export CSV button in the Export data section.

   

   Here’s an example of a grades CSV from an Assigned assessment:

   

See also

• Exporting a list of course grades
• Downloading student work

You can view analytics about scores and responses as well as question and distractor quality on any assessment that uses Crowdmark’s multiple choice bubble sheet. After booklets have been matched to students and the assessment has been graded you can view the analytics by navigating to the Results page of your assessment and clicking the Multiple choice analytics button.

Scores

The Scores page displays statistics for the total scores on the multiple choice component of your assessment. You can also download the data as CSV files to analyze question and student responses.

• Number of students: The total number of students matched to a graded assessment.
• Average score: Mean total score
• Median score: Middle score when the total scores are sorted from lowest to highest
• Lower quartile: Middle score of the lower half of scores when sorted from lowest to highest
• Upper quartile: Middle score of the upper half of scores when sorted from lowest to highest
• Highest score: Highest total score
• Lowest score: Lowest total score
• Standard deviation: Measures the amount of variation in the total scores

  where n is the number of students, xi is the total score for student i and x is the average total score on the test

Question data export

From the Scores page, you can download a CSV file of data for each question. The following statistics are available in the export:

• Percent answered correctly: Percent of students who answered the question correctly
• Class: Percent of all students who answered the question correctly
• Top 27%: Percent of students who answered the question correctly in the top 27% of scores when total scores are sorted from lowest to highest
• Bottom 27%: Percent of students who answered the question correctly in the bottom 27% of scores when total scores are sorted from lowest to highest
• Average Score: Mean score on question
• Standard Deviation: Measures the amount of variation in the question score
• Point biserial: Measures how well the question discriminates between high and low scoring students. Point biserial less than 0.2 should be rejected or improved by revision
• Response Frequencies: Percent of students who selected each response (A, B, C, D, E, or x for no response)
• Answer Key: Correct response or responses to question
• Non-Distractors: Response(s) to question not selected by any students.

Student data export

From the Scores page, you can download a CSV file of data for each student. The following statistics are available in the export:

• Scores: Score on each multiple choice question
• Total Score: Total score on all multiple choice questions. Total score has a minimum value of 0. If the sum of the scores produces a negative value, Crowdmark will record it as 0.
• Responses: Selected response for each question (A,B,C,D,E, or x for no response)

Questions

For each multiple choice question on your test, Crowdmark displays a bar chart showing the frequencies of each response and of no response selected (denoted “x”) on the Questions page. Hovering over one of the green bars will show you the percentage of students who selected this response. The correct response for the question is typed in bold and the corresponding bar is highlighted in darker green.

Diagnostics

The Diagnostics page summarizes your multiple choice questions in terms of difficulty, discrimination and distractors. Often referred to as item analysis, these diagnostics give you insight into your individual multiple choice questions as well as some test-level statistics to help you determine the quality of your test. Crowdmark highlights your most difficult and easiest questions, questions that are most and least effective at discriminating between your best and worst performing students and questions that include functional and nonfunctional distractors.

• Difficulty: The difficulty of a question is determined by computing the percentage of students who answered the question correctly.

  Crowdmark classifies questions as easy when more than 70% of students answer correctly and difficult when less than 30% of students answer correctly.

• Discrimination: The discrimination of a question refers to the correlation between student responses to a particular question and total scores on the test. Crowdmark uses the Point Biserial coefficient, denoted by rpb, to calculate the discrimination of the correct response to each question using the formula:

  where M1 is the mean test score for the group who answered the question correctly, M0 is the mean test score for the group who answered the question incorrectly, sn is the test standard deviation, p is the proportion of students who answered the question correctly and q is the proportion of students who answered the question incorrectly.

  A positive point biserial coefficient for a correct response indicates that those who scored well on the whole test answered the question correctly. Generally, the correct response for a point biserial greater than 0.40 is a very good discriminator and less than 0.2 should be rejected or improved by revision.

  Point biserial correlation coefficients can also be calculated for each incorrect response (distractor). A question should have a negative point-biserial discrimination for each of its distractors. This means that the incorrect responses were selected by the weaker students.

• Distractors: Distractors are the incorrect responses to a question.

  Crowdmark classifies questions as poor distractors poor when an incorrect response to the question was selected by less than 5% of students or the incorrect response was selected by top performing students.

  Crowdmark classifies questions as having good distractors when all incorrect responses are chosen by at least 5% of the students and incorrect responses are selected by weak students.

High-level statistics

• Distractor quality: Distractors are the incorrect responses to a question. For example, if the correct response to a question is B, then the distractors would be A, C, D and E. A good distractor is one that was selected by students with low achievement. Crowdmark measures the distractor quality of the overall test by calculating the percentage of good distractors on the test. Crowdmark classifies a good distractor as having a frequency of response of at least 5% and discriminating towards students with a lower total score. Crowdmark uses the point biserial coefficient to calculate the discrimination correlation of each distractor.

• Reliability coefficient: Measures whether the test as a whole discriminates between higher and lower performing students. Crowdmark uses the Kuder-Richardson 20 coefficient (KR-20) to calculate the test reliability, which we will denote by r in the formula below.

  

  where n is the number of questions, pi is the proportion of students who answered question i correctly, qi is the proportion of students who answered question i incorrectly and sn is the test standard deviation.

  The KR-20 coefficient typically ranges between 0 and 1 but it is possible to obtain a negative value with small sample sizes. The closer to 1 the more reliable an exam is considered because its questions do a good job consistently discriminating among higher and lower performing students.

  The interpretation of the KR-20 depends on the purpose of the test. Tests intended to distinguish between students who have mastered the material and those who have not should have a reliability coefficient of 0.5 or higher. A test with a KR-20 value less than 0.3 is considered poor. If the purpose of the test is to ensure that all students have mastered essential concepts, then the KR-20 value should be close to 0.

Sources:

• Distractor Efficiency in an Item Pool for a Statistics Classroom Exam: Assessing Its Relation With Item Cognitive Level Classified According to Bloom’s Taxonomy
• Kuder–Richardson Formula 20 on Wikipedia
• Interpreting Exam Performance: What Do Those Stats Mean Anyway?
• Statistical Analysis of Multiple Choice Exams
• Item Analysis for Criterion-Referenced Tests

After the grading is complete, an Instructor or Facilitator can return student work with scores and feedback online. To do this:

1. Navigate to your assessment and click Results in the sidebar.

    

   

2. Click the Return grades button.

    

   

This will send an email to each student containing a link to their score page, which includes their work, scores, feedback, and any other information you’ve chosen to include.

You can send a message to your students and choose whether or not to include the class results graph by clicking Edit beside Student view settings. You can change these settings anytime and students’ score pages will be updated automatically.

For more information about the score page, see What do students see?

When you send grades to students, you have the option to include a message that will appear on their score page when they view their results. The message can be edited anytime and will be automatically updated on all score pages.

Adding a summary message

1. In your assessment’s Dashboard, click on Results in the sidebar.

   

2. Click the Pencil icon beside Assessment Summary Message.

   

3. Enter a message and click Save. 

    

   

If some students will be writing an exam remotely (with accessibility services or in a distance education scenario, for example), you can enable the remote session feature within an existing assessment. This feature provides a proctor with the resources to administer the exam and upload student work to Crowdmark. Creating a session and inviting a proctor will enable them to:

• Sign in to Crowdmark
• View the session details (date, location, participating students)
• Download a QR-coded exam booklet for each student (a page is added to each booklet with the student and course information)
• Scan and upload completed exam booklets to Crowdmark

You will be able to track the proctor’s progress as they complete these tasks. Since each student is assigned to a specific booklet, there is no need to match the proctored assessments to students after the pages are uploaded.

Enabling remote exam sessions

Remote sessions need to be enabled per-assessment.

1. Navigate to your assessment’s dashboard.
2. Click Assessment settings.
3. Click the Enable remote exam sessions toggle. A Sessions link will appear in the sidebar. This is where you can set up a remote session.

Creating a remote session

After the remote sessions feature is enabled, you may create a session.

1. Click Sessions in the sidebar. The sessions table will appear.

   

2. Click Add session. A window with setup options will appear.

   

3. Enter the Date and time your session will take place (if it doesn’t have a set date, you may choose a date range). Enter the Location (optional).

4. Click Add session. The Session details page will appear.

5. Under the Students section, click Add students.

6. Enter the name or email addresses of one of the students who will participate in the session. As you type, Crowdmark will narrow down the class list. Select the correct student.

7. Repeat for each student, then click Update to add the students.

Inviting a proctor

Once your session has been created and has students, you can invite the proctor to administer the exam.

1. Ensure that the session details page is open. Under the Proctor section, click Invite a proctor.

   

2. Enter the proctor’s email address, a note with any additional instructions you would like to include (optional), and your contact information (optional).

   

3. Click Invite proctor. An email will be sent to the proctor inviting them to sign in to Crowdmark to access the session details.

What will the proctor see?

• The invitation email will include a button for the proctor to accept the invitation and view the session page. If the proctor doesn’t have a Crowdmark account, they will be prompted to sign up before accepting the invitation.

  

• After accepting the invitation, the proctor will be directed to the session page.

  Note: You can also access this page by navigating to the assessment’s Sessions page, then clicking the proctor’s name in the “View as proctor” column.

• Each booklet the proctor downloads will have a QR code on each page (like normal Crowdmark exam booklets), but the booklet will be assigned and matched to a specific student. Crowdmark will add a page to the beginning of each booklet with student and course information to ensure that proctors distribute booklets to the correct students:

• There is a link to documentation for proctors in the right sidebar of the session page: Proctoring a remote exam session.

When you use Crowdmark booklets for an assessment, can automatically match cover pages to students using Optical Character Recognition (OCR). You must leave a blank space on your cover page for Crowdmark to add a region for students to write their information. Your student data must include First Name, Last Name (or just Name) and Student ID columns. Student ID must be numeric. See Managing students in a course for more information about student data.

How it works

1. Leave a blank space on your template’s cover page. Leave a blank space between 9 cm (3.5 inches) and 18 cm (7 inches) down the cover page for this region in addition to the 3.8 cm (1.5 inch) space for the QR code.

   See examples below of the Crowdmark-generated QR code and automated matching region (shown in blue). Click the images to enlarge.

   

2. Enable automated matching when setting up your assessment template. When you upload your template you will have the option to enable automated matching. Select this option. You may customize the Student ID label to be more meaningful to your students. Upload your template and finish the assessment setup process.

   

3. Administer the assessment and upload the completed booklets. Students can write their information in uppercase or lowercase writing or a combination of both, with one character per box. They should print each character clearly and within the boxes. If a student’s name has more characters than the boxes provided they should only write one character per box and continue writing their name to the right of the boxes.

4. Crowdmark will automatically match the cover pages that have been uploaded. You can monitor the progress on the Match booklets page.

5. Match any assessments that could not be matched automatically. Use the Next unmatched button on the Match booklets page to navigate through and match any remaining cover pages.

If you use an LMS (Blackboard, Canvas, Brightspace, Moodle, etc.)

In most cases, Crowdmark can match automatically based on the student information imported from your LMS. Sometimes, depending on your LMS and your institution’s configuration, Crowdmark may not be able to access student ID numbers. You can verify whether or not Crowdmark has access to the ID number by navigating to the Students page in your course and seeing if there is any data under the Student ID Number column.

If you use Blackboard or Moodle, the integration technology does not allow Crowdmark to import student numbers. If you use another LMS and the student number is not imported, please contact Crowdmark Support; we may be able to work with your institution’s IT staff to import student numbers in the future.

As a workaround, you may update student information manually with a CSV file. See Managing students in a course for more information.

Troubleshooting automated matching

Match rates on assessments are on average above 90%. This means 90% of student information was recognized by our OCR system and able to be matched automatically to students in your class. However, sometimes there are issues that cause match rates to be lower. Here are some reasons match rates might not be as high as expected:

• Student handwriting issues. If students’ writing is unclear, too light on the page, if letters are scratched out, or if the writing crosses lines in the auto-matching info grid, the OCR system may not be able to match.
• Alphanumeric student ID. Student IDs are required to be numerical in order to be recognized by OCR. If they contain letters, the possibility of matching is decreased significantly. If your student IDs contain letters, you may wish to import custom ID numbers students can use on Crowdmark assessments.
• Lack of sparsity in student information. OCR requires sparsity to recognize information. This means that widely-varied characters are more easily recognized than sequential or similar characters. If student names or numbers are sequential, or contain repeating characters, matching is less likely.
• Scan quality. Higher contrast is better, as it will be easier for the system to differentiate between the characters and the space around them. Mottled scanning, scanning in black and white, or low-contrast scanning will have a negative effect on match rates. See this document on scanning assessments for more details.
• Incorrect scanned area. If the edges of scanned pages are not clear, or if there is additional space around the pages, OCR will not be able to complete matching. See this document on scanning assessments for more details.

Crowdmark can automatically grade up to 200 multiple choice questions in an administered assessment. If you enable multiple choice on your assessment, Crowdmark will add up to two 100-question bubble sheets with responses A–E to the end of each student’s booklet. Multiple answers to a question is supported.

Benefits of using Crowdmark’s multiple choice bubble sheet workflow include:

• Fully auto-graded
• Easily edit student responses or errors
• Compatible with standard printing and scanning
• Compatible with standard 8.5×11 or A1 paper
• QR-coded pages mean students are easily matched to their work

How it works

1. Create your assessment template. Crowdmark requires at least one page (a cover page) in the template. You can include the multiple choice question prompts in your template or leave them out and print them outside of Crowdmark. Crowdmark doesn’t currently offer a way to “link” the question prompts to the bubble sheet but we recommend including the questions in the template so students can reference them after grades are returned.

2. Enable multiple choice when setting up your assessment template. You will have the option to enable multiple choice, by choosing the number of bubble sheets you’d like to include. When the time comes to define the locations of your questions you do not need to label the multiple choice questions in your template.

    

   

3. Set default scoring options. From the assessment’s Dashboard, navigate to the Questions page in the sidebar. Click the Multiple choice settings button. Here, you have the option to:

   • Edit the default points value for a correct response. By default students receive a score of 1 for a correct response.
   • Edit the default points value for an incorrect response. By default, students receive a score of 0 for an incorrect response.
   • Select how a question with multiple answers is interpreted. The options are: Students need to select ALL of the correct answers (e.g. A and B) and Students need to select ONE of the correct answers (e.g. A or B).

   These settings apply to all multiple choice questions in this assessment. Question-specific changes in the answer key (step 4) will override these settings.

4. Complete the answer key and customize question points. On the Questions page, click on “Answer key” beside the MC questions. Click the bubbles to select the correct answer(s) to each question. You may select multiple answers to a question. If the page shows more questions than you have, leave the extras blank; Crowdmark will only grade questions you have selected answers for. You may use the input boxes beside each question to change the number of points students will receive for a correct answer.

   

5. Print, administer, scan, and upload the completed booklets. Students can use pen or pencil to fill in the bubbles. Use the edge detection setting on your scanner to ensure that no black or grey background is included around the edges of the scanned images.

6. Crowdmark will automatically grade the bubble sheets that have been uploaded. Students receive the default scores (set in step 3) for each correct and incorrect response. The MC question/page score is the total for all multiple choice questions on that page.

   

7. Resolve any unknown responses. If Crowdmark is unsure about a response on a page or a student has not answered a question, the page will be marked with “unknown response” (and ungraded). This may happen if a student has mistakenly filled in a bubble but has attempted to cross it out or fill in a different bubble.

   On the Dashboard, under the Grading heading, you will see the MC question blocks. If there are unknown responses, click MC.

   

   You’ll see the first unresolved page in the grading interface. The unknown response will be highlighted in yellow with a “?”. Clone the evaluation and click the bubble that the student selected to resolve the response.

   

What do students see?

Students will see a score beside each answer. The correct response to each question will be highlighted in green.

Troubleshooting bubble sheet issues

If you find there are errors with the bubble sheet scans or auto-grading, here are some things you can verify:

• Errors can be caused by bad can quality or a visual artefact disrupting the detection of the multiple choice region (a line, shape, or writing/drawing).
• It’s important to make sure the black triangles surrounding the bubble area are clear and present.
• If a line is appearing, your scanner may need to be cleaned.
• If the area where the staples were cut off is scanned in black it may cause a problem with detection. To solve this issue, ensure that your scanner makes any extra space around the scan white.

If the multiple choice area cannot be detected at all, Crowdmark treats the page as any other page in the template and you may grade it manually. You can enter a score in the sidebar and add annotations and comments to the page. Please contact Crowdmark Support with any other questions.

In Crowdmark, a course contains your student roster, grading and administrative team and assessments. After signing in to Crowdmark, you will see the My Courses page where you can view existing courses and create new ones.

Creating a course

1. From your My Courses page, click the Create a course button. A modal window will appear.
2. Enter the name of your course. If your course has sections, check the This course has sections checkbox.
   

   

3. Click Create course.
   

   

Next steps

• Enroll students in a course
• Add team members to a course

From the Assessment setup page or the Settings page, you may export the assessment’s entire comment library or comments from a specific question to a CSV file. This file can be used for analysis or imported to other assessments in Crowdmark.

To import and export comments from a CSV navigate to the Assessment setup page, or the Settings page.

You can create two kinds of assessments in Crowdmark: Administered and Assigned.

The Administered assessment allows an instructor to create paper exams, tests, or assignments that are written by students and then scanned into Crowdmark for evaluation.

The Assigned assessment empowers an instructor to deliver assessments to students electronically with a due date and provides students with methods to enter or upload their responses for evaluation on Crowdmark.

All students and team members are included in an assessment by default but you can include or exclude specific students, sections or team members.

Creating an assessment

1. From your course page, click Create assessment
   
2. Enter a name for your assessment in the Assessment title field and click Next.
   
3. Choose the type of assessment you wish to set up. To learn about the difference between Administered (in-person) and Assigned (online) click here.

Creating an Assigned (online) assessment

1. Select Assigned (online)
2. Choose whether students will submit as an individual or a group. Click here to learn about group assessments.

3. You can use the Previous arrow to go back to any of the previous steps to change your selections.
4. Confirm you selections by clicking Finish. This will take you to the Assessment setup page.

5. From the Assessment setup page you can create questions, schedule the assessment, manage the student and team rosters, and manage the comment library.

Next steps

• Defining questions
• Using Multiple choice questions in an Administered assessment
• Managing students in an Administered assessment
• Managing the comment library

Team members include co-instructors, TAs, graders, anyone who will help with printing, scanning or grading assessments in your course. A description of each role can be found here: What is the difference between team member roles?

Adding team members

From the Course team page, team members can be added with their email addresses. After you enter the email addresses you can select which role each team member should have.

To add team members:

1. In your course, click Course Team in the sidebar.
   

   

2. Click the Add by email button in the upper right corner. A modal window will appear.
   

   

3. Type or paste your team members’ email addresses in the field. Click the Continue button.
   

   

4. You will see a list of the team members you have entered. Under the Role column, select the role each team member should have. To remove a team member, click the x. For more information, see What is the difference between team member roles?
   

   

5. Click the Add team members button. Your team members will be added to the course and will each receive an invitation email.
   

    

What do team members see?

Course invitation email: 

After you add a team member, they will receive an email from Crowdmark. The Go to course button will take the grader directly to the course. If the team member is a new user, they will first be prompted to set up an account.

Assessment invitation email:

Depending on the team member’s role, they may have access to all assessments (Instructors, Facilitators) or they may only have access to assessments where they have been invited (Graders, Uploaders). In order for a Grader or Uploader to have access to an assessment, they must be included in that assessment’s grading team. You can see the grading team settings in the assessment by clicking Team, then Manage team from the side bar, or by selecting Customize under Team from the Roster panel on the Assessment setup page. This will open the assessment team roster. 

Clicking the checkboxes allows you to select specific team members to include/exclude or use the tabs include/exclude anyone with a specific role. You can also use the top left checkbox to include or exclude all team members in that tab, or use the search function to find a team member. Once you have made your selections, click Save.

Once added, team members will receive a similar notification email each time they are included in an assessment.

Removing team members from the course

From the Course Team page, a team member can be removed by clicking the x beside their name. This will remove their access to the course and its assessments but all of their past grading and data on Crowdmark will be preserved.

Updating a team member’s role

If you would like to update a team member’s role so they have different permissions, navigate to the Course Team page, search for the relevant team member and use the drop down arrow beside their current role to select a new one.

After your assessments are uploaded, you must match their cover pages to students. In Crowdmark, this can be done anytime after uploading student work: before, during, or after grading. 

Crowdmark booklet workflows require a separate cover page for matching, which will not appear in the grading interface. Ready to grade assessments do not require a separate cover page, and all pages appear in the grading interface. In either case, a booklet must be matched to a student in order for that student to receive their grades and feedback. 

Crowdmark booklets offer an automated matching option that uses Optical Character Recognition (OCR) to scan your students’ cover pages and match them with existing student information. For more info on this process, see our document on using automated matching.

Crowdmark also offers Exam Matcher, an app which enables you to match students while taking attendance at the beginning of an exam that’s using QR-coded Crowdmark booklets. Read more about Exam Matcher.

Matching assessments to students

1. 1. There are two ways to navigate when beginning the matching process: you can select Match booklets from the side bar menu, or click the tile on the Assessment setup page.
   2. On this page you will see an overview of the matching progress including the percentage of booklets automatically matched, manually matched and unmatched.

      

   3. If you’ve opted to use automated matching, the process will begin in the background, or you can click Run automated matching to restart the OCR matching process. Depending on your class size, this may take a few minutes.
   4. If you’re not using automated matching, or if you have some students who could not be matched by OCR, select manually match booklets. A cover page and search field will appear. Locate the student’s information on the cover page and begin typing it into the search field. The list of students will be filtered as you type.
      
   5. Once you’ve found the student, click on their email address to match them. The page will automatically move to the next unmatched booklet. To stop the page from moving foward, you can turn off the Auto-advance to next booklet toggle. You may also use the Next unmatched button to skip a booklet or move to the next unmatched booklet manually. Change the filter to All to search for a specific booklet number, return to the previous page or move to the next booklet sequentially. Changing the filter back to Unmatched will allow you to continue matching in sequence. Repeat until all cover pages are matched.

Matching labels

Each match will be labelled with the student’s name and email address. You’ll also be able to see whether the page was matched automatically, manually, or using Exam Matcher. If the page was matched using Exam Matcher, you’ll be able to see the room number and proctor information. You can filter by how a page was matched using the dropdown menu.

Labelling blank or extra booklets

Below the matching search field you’ll see a checkbox that allows you to mark booklets as Extra. Use the checkbox to indicate if an exam booklet does not need to be matched.  Extra booklets will be skipped over when you click Next ungraded, and will be labelled as Extra in the Overview grid.

Fixing a matching mistake

If a student has been incorrectly matched, set the filter to All and use the Search to find that student’s booklet by name or booklet number. Click the trash can icon to un-match that student. The page will stay on the unmatched booklet, allowing you to match another student. Changing the filter back to Unmatched will allow you to continue matching.

Keyboard shortcuts

You can save time by using keyboard shortcuts. While matching, a list of shortcuts can be found by clicking the Keyboard shortcuts link below the search bar.

Matching multiple students to an assessment

Some assessments involve student collaboration: two stage exams, lab reports, and group projects are some examples.

It is possible to match any number of students to a cover page by turning off the Auto-advance to next booklet toggle. This will prevent the page from automatically switching after you’ve matched a student. Once you’ve matched all of the students to the page, turn the toggle back on, or click Next unmatched to advance to the next booklet.

Each student matched to an assessment will receive their own copy of the graded assessment once grades are returned.

An instructor or facilitator can duplicate any assessment that they have created in Crowdmark, from any course. This makes it easy to re-use and re-purpose assessments from previous courses. To duplicate assessments,

1. Use the Search function at the top of the dashboard to locate the assessment you wish to duplicate. In your My Courses page you can search for a course, and in the course dashboard you can search for an assessment.

Instructors and Facilitators can create rubrics in advance of grading by creating shared comments with points in the Comment library. When a comment is shared, it will appear in the Comment libraries and comment suggestions of all Graders, Facilitators, and Instructors on the team. Team members can then drag and drop the comments onto student work to apply the feedback and points to the question.

Setting up a rubric

You have two options for setting up a rubric in the comment library: you may set it up directly within the library in the grading interface, or you may fill out and import a CSV of comments to the library.

To set up a rubric within the comment library

1. Open the Comment library.
2. Click the New comment button in the upper right corner. A text box will appear.
3. Click the Share with team toggle above the question content. The comment will be shared.
4. Enter the comment text and points and click Save.
5. Repeat with each rubric item.

Two-stage exams are a tool used across higher education to promote collaboration, comprehension and assessment as learning. In this multi-step exam format, students write the exam independently, then write the same exam again as a group.

Benefits of a two-stage exam format for students include:

• Immediate feedback on their first round of testing
• An opportunity to learn from their peers
• In many cases a higher overall grade

For a detailed overview of these benefits, check out our blog. In Crowdmark, two-stage exams can be conducted using Assigned or Administered assessments, or a combination of the two.

To create a two-stage exam

1. Start by creating your first assessment. This will be the individual evaluation.
   • If your students are completing the exam in-person, you can use the Administered assessment workflow, or use the Assigned assessment workflow for hybrid submission.
   • If your students will be writing online, use the Assigned assessment workflow.
2. Once the first assessment is created, you can create the second assessment. This assessment will be set up for groups.
   • If you’re using the same format as the first exam, you can use the duplication function to save time creating your exam content. See info on duplicating Assigned assessments and Administered assessments in our Help Center.
   • If you’re using an Assigned assessment, make sure to select the Group assessment option in the assessment setup.
   • If you’re using an Administered assessment, you can use the toggle to match multiple students to a booklet.

Once you have grades returned in Crowdmark, you’ll find a performance report in your course dashboard that will show your progress across assessments. To view your performance report:

1. Click into your course, and click the Performance report button.
2. Your performance report will look something like this 
3. The line graph section shows your progress across any graded assessments in the course. If your instructor has included class averages, they will also show in a separate line.

   

4. If you scroll down you’ll see your top and bottom scoring questions in the course. Clicking in to any of these will take you directly to the graded assessment, where you can view feedback from your instructor.

   

By default, your graded work can only be viewed by you and your instructor while signed in to Crowdmark. If you wish to share your grades on an assignment, test, or exam you can generate a link that can be shared with others. The link is public, meaning that anyone who has the link can see the page without signing in to Crowdmark, but the URL is a long, random string of characters that would be impossible to guess.

Sharing your results

1. Open your graded assessment.
2. Click Get shareable link. The URL will be copied to your clipboard so you can paste it wherever you’d like to share it. You can access the URL again anytime by repeating these steps.

Both you and your instructor have access to a log of all actions you have taken when submitting an assignment. If something has gone wrong or if you are not sure when your assignment was submitted you can look at the log.

Viewing your submission activity log

1. Go to the assignment list in your course. Click the Submitted or Not submitted link under the Status heading. The submission log will appear.

   

2. You will see a list of actions including the time and name of the person who performed each action. This is an example of a typical submission:

   

   A normal submission includes all of the actions below:

   • Viewed assignment page
   • Started timed assessment (Timed assessments only)
   • Student was added to a group (Group assessments only)
   • Uploaded 2 pages to Q1:image1.jpg, image2.jpg (Image/PDF upload questions only)
   • Answered Q2 (Text and Multiple Choice questions only)
   • Clicked submit button
   • Submission time logged
   • Submission processing completed

   Submission time logged is the official time that you submitted. Submission processing completed means that your files were successfully uploaded and processed.

If something goes wrong during the upload you will normally see an error message and an Upload failed entry will show up in the log (your instructor will be able to see it as well). A list of errors and their resolutions is available on the Completing and submitting an assessment page.

Viewing file details

MD5 image hash

Crowdmark logs the MD5 hash (the file’s unique identifier) for each file that you upload to an assessment. What that means is if you submit a file, then submit the same file again, you’ll see the hashes on the two submissions are identical. The MD5 hash is built into the file when it is created on your device or computer. If you alter the file by changing the format, editing the image, or even re-saving the same file, the hash will appear different from the original image hash.

Here’s an example of what MD5 hashes look like in your submission log. By default, the hashes won’t be visible. You can click the Show file details button at the top of the log to show the hashes beside each file.

There are two ways to sign in to Crowdmark from the Sign In page:

1. With an email address and password.
2. Through your school’s login (if your school supports this).

If this is your first time signing in to Crowdmark, see Claiming your account (signing in for the first time).

To access your sign-in options

1. Go to the Sign In page.
2. Choose your institution from the dropdown menu.

Signing in through your school

Some schools offer the option to sign in using your school credentials. If you have this option, we recommend that you continue using it.

1. If your school offers an integrated sign-in, you’ll see a page with the school logo and a sign in button. If you do not see this, your school is not yet integrated with Crowdmark and you will have to sign in using your email address and password.

2. Click Sign in with [your school’s LMS] (“LMS” is whichever system your school uses). You will be prompted to enter your school username and password. Depending on the system your school uses, you might also have to enter an email address and verify it the first time. Crowdmark will take you through this process.

   

3. After you have signed in successfully, you will see a page titled My Courses, which is your portfolio of assessments on Crowdmark.

   

Signing in using your email and password

If your school does not have an option to sign in using your school credentials, you’ll sign in to Crowdmark using a user email and password selected by you.

1. Enter your Email and Password and click Sign in. You will see your My Courses page.

If this is your first time signing in to Crowdmark, see Claiming your account (signing in for the first time).

Troubleshooting the sign in process

If you receive an Invalid password error message, you may not have set a Crowdmark password, or you may not have finished the Claim account workflow. Click Forgot your password? to set a password for you Crowdmark account.

If you enter an incorrect password too many times, you’ll receive a message that your account has been locked. To unlock your account, click Unlock account on the sign-in page and follow the instructions in the email you receive. Do not click the unlock button more than once, as the email you receive contains a unique one-time-use link which will be invalidated by multiple clicks.

If you sign in to Crowdmark and cannot see your courses or assessments, there are several possibilities:

• You’re signed in using a different email address than what has been enrolled in the course or assessment. Try signing in with any other email addresses you have, and check with the instructor to verify.
• You haven’t been invited to or enrolled in that course or assessment yet. Check with the instructor to verify.
• Your LMS email address has changed, and your single sign on is sending you to a new account. Check your local LMS settings, or try signing in to Crowdmark directly using your email address and password. See above for password issues.

If you have confirmed you have multiple Crowdmark accounts, merging the accounts will allow you to use the email addresses interchangeably. If you still cannot access your courses, please contact support@crowdmark.com.

If you are an Instructor or Facilitator in the assessment, you can generate tokens for your team of proctors by following the instructions below.

For Instructors:

From the settings page in your administered assessment, click Generate a single use token. 

Take attendance and match students to booklets at the same time using any mobile device. If you have a team of proctors, you can generate a unique sign in token for each team member. These tokens will allow anyone with a mobile phone or tablet to conveniently match assessments with students.

Getting started

1. Sign in using the link and unique token provided to you by your instructor.
2. Search for and select a student by using their name, email or ID number. 
3. Scan the QR code on any page of the student’s exam booklet.
4. Repeat for the next student.

• Exam Matcher: How it works

Crowdmark allows for a high degree of flexibility in timing when assessing students remotely. With students all around the world, you may often see students in multiple time zones writing a single test, exam or assignment. Using Crowdmark you’re able to choose the best set up for your students in every assessment.

How do time zones work in Crowdmark?

Time zones in Crowdmark are recorded in UTC and displayed in the locally set time zone on your machine. If you’re building an assessment in British Columbia and you set the due time for 12PM PST, it will show to students in Ontario as 3PM EST. This applies for time zones around the world.

Time zones in Crowdmark also adjust automatically for Daylight Savings, according to your machine’s setting. For example, Saskatoon and Winnipeg are both in Central Time, however Saskatchewan does not use Daylight Savings, whereas Manitoba does. That means in the summer, Central Standard Time and Central Daylight Time show up as two distinct time zones. When time changes happen in Spring and Fall, due times adjust automatically. For example, if you are working in EST and set a future due date after the time change, it will display as EDT on the assessment

Synchronous and asynchronous assessments

When you’re building your Crowdmark assessment, there are two distinct ways to set up the timing: synchronous and asynchronous. Synchronous assessments are when all students write at the same time, and asynchronous assessments are when students write at different times, usually at a time of their choosing. Each of these options has a procedure for set up as well as pros and cons, which we will outline for you in the following.

Synchronous assessment

A synchronous assessment means all students are writing together. This most closely resembles an in-person testing scenario, where everyone starts and ends at the same time.

Here are the steps to set up a synchronous assessment:

1. Begin the workflow to create a new assessment or duplicate an existing assessment. Follow the steps to the Setup page.

2. Complete your assessment question setup in the Questions tab. Click Schedule your assessment.

3. Set the distribution time and due date of your assessment in the Schedule tab. The distribution time should be the time students will be starting their assessment. The due date is the time students will be handing in their work. Do not use the Time limit toggle. Click Schedule distribution.

Congratulations! You’ve now set a synchronous assessment, where there is one set start and end time for everyone. If you have students who require extra time, you can change due dates on an individual basis using the instructions here: Accommodation options for assigned assessments

To set up multiple synchronous assessments for students in different time zones, check out the instructions here: Managing sections or multiple versions of an assigned assessment

Asynchronous assessment

Asynchronous assessment means students write at different times, usually at a time of their choosing. This would be closest to a typical homework or lab assignment, or a take-home test. It is the most flexible option for students who have a large degree of variation in their individual timing requirements.

In Crowdmark, you have the option to use a Timed assessment to give students flexibility of choosing when they’re going to write, while still maintaining a limited window of time where they sit down to work. An example would be a 2 hour exam, within one week. You can also use this option to set up assessments where students don’t have a particular due date, but can take the timed test at any time over the course of an academic term. To set up a Timed assessment, follow the instructions here: Creating a timed assessment

You can also set up an asynchronous assessment where a student has a window of time to work on an assessment, but is able to access it and come back to it at any time before the due date passes. This would be closest to a typical homework assignment, and can be set up as a typical assessment with a standard due date, without using the Timed assessment function.

When using asynchronous assessments, the needs of students with academic accommodations are easily met. An instructor can change due dates (for someone who needs an extra day or two), time to complete (for someone who needs extra time on their Timed assessment window) and late penalties on an individual basis using the instructions here: Accommodation options for assigned assessments

Notification email

After you click Distribute or if distribution happens automatically by scheduling, an email will be sent to every student. It will look like this:

Even if a student does not receive the email, the assessment will be visible when the student signs in to their Crowdmark account.

Viewing the assessment

The Go to assignment button points to the student’s assessment submission page (the student will need to sign in if they are not already signed in). Here, the student will see:

• Assignment description
• Assignment questions (students will not see question versions, if applicable)
• Due date
• Lateness penalty (by clicking on the due date at the top)
• Countdown timer

When there are less than 15 minutes left to submit, students will see a warning message reminding them to upload their work as soon as possible.

Submitting the assessment

How students answer each question depends on the response type that you have specified for each question:

• For Image/PDF file responses, students need to complete the question on paper, in Word, Google Docs or a note-taking app, then scan or photograph their work. The files can be uploaded to Crowdmark by browsing for or dragging and dropping the pages in the drop zones under each question. Pages can be rotated, reordered, and moved between questions if needed.
• For Text or Multiple choice, students can input their responses directly in Crowdmark.

Confirming the submission and resubmitting

This is what students will see after submitting their assessment. They will have the opportunity to double-check their submissions and resubmit.

Submitting late

Once the due date has passed, the submission is locked in and students cannot resubmit. A student may only submit once (for the first time) after the due date has passed. Because grading can start immediately after the due date has passed, this ensures that students do not overwrite their submission after it has been graded.

If a student has not submitted yet and views the assessment after the due date, a popup windown will appear informing the student that their submission will be late and what the penalty will be (if any).

Viewing a specific student’s submission page

1. In your assignment’s Dashboard, click on Students in the sidebar.

   

2. You will see a list of students. Locate the student, then under the three dot menu on the right, click Submission.

   

3. You will see the student’s submission page. If needed, you can submit on behalf of a student from this page as well. To exit, click the x in the top right corner.

   

Help for students

When students are viewing the assessment, they will see a Help link linking to these detailed instructions: Completing and submitting an assessment.

If you notice that a page of student work has been uploaded in the wrong orientation it can be easily fixed during grading.

1. Click the sidebar icon in the lower left corner. This will open a drawer with a thumbnail view of the pages.

    

2. Click the Rotate icons on the page you wish to rotate.

   

Tags provide a flexible way to organize and bookmark student work and evaluations with your team members. While grading, you can use a tags to bookmark exemplary answers, share questions that require another look, or to indicate suspected cheating. Students cannot see tags.

Adding a tag

1. Navigate to a question in an assessment as you would if you were going to grade it.
2. Click the Tag button in the sidebar (or enter the keyboard shortcut “t"). The tag input box will appear.
3. Enter a name for the tag and click the + button. The new tag will appear below.

Viewing evaluations with specific tags

You can view all evaluations with a specific tag or group of tags using the filter tool. See Filtering evaluations to learn more. Tagged booklets require an evaluation on the page to be included in the filter.

Comments can be managed through the comment library tool, which can be accessed through the book icon in the grading toolbar.

The comment library shows all comments you have left on a question. Graders can see their own comments and shared comments, and Instructors and Facilitators can see everyone’s comments. Using the comment library, you can view comments filtered by version, as well as view all shared comments for a question.

Benefits of using the comment library include:

• Once a comment is in the library, it can be used to grade that question through the entire assessment.
• If you select a comment then edit the content, it will save as a new comment.
• Comments from randomized questions are labelled by version, and they are available to be used across any version of that question.
• Comments can be edited once to change every instance used across the assessment
• Instructors and Facilitators can also set up comments in the comment library or set a rubric using comments before grading starts.
• A comment can be shared with the grading team by turning on its Share with team toggle. Shared comments will appear in every grader’s comment suggestions.

Searching for comments

To find a comment in the comment library, use the Search bar in the top left. All comments containing that keyword will appear, along with labels for which question and version they belong to.

You can also search for a comment while grading: when you begin to type comment content, auto-suggestions from your comment library will appear below. If you’re grading a question with versions, comments that apply to the version you are grading will appear first, followed by comments from other versions.

Comments from all versions will be available to be used. Duplicate comments across versions will display in each version, and will be linked automatically in the library to preserve bulk-editing.

Filtering evaluations by comment

Above each comment you will see the number of times it has been used. Grading teams can click on this number to show all evaluations using the comment using the filter tool. If the toggle to Allow graders to see each other’s evaluations is turned off, only Instructors and Facilitators will have access to this comment filter.

Bulk editing comments

If a comment has been used multiple times, you can make changes to all instances of the comment by using the comment library. Graders can only see and make changes to their own comments, Instructors and Facilitators can see and make changes to everyone’s comments. Duplicate comments automatically revert to the existing comment in the library, so you won’t need to worry about having to edit more than one. It will be important, however, to keep track of minor edits that have saved as a new comment. In an assessment with question randomization where there are linked duplicate comments across versions, you will have the option of editing the comment for the current version, or for all versions.

1. Select the comment you wish to edit.
2. Click the Edit buttton below the question content.
3. Make your changes and click Save. All instances of the comment will be updated.

Reordering comments

You can reorder comments to customize how they are displayed in the comment library as well as in the grading toolbar. To do this, drag a comment up or down the list of comments in the left sidebar of the comment library.

Adding a comment before grading

If you have a rubric or example feedback prepared in advance, you can create comments and associate them with point values without grading student work.

1. 1. Select the question or version you’re working on, and click the New comment button in the upper right corner of the comment library. A blank comment will appear.
   2. Enter your comment text, point value (optional) and click Save. Your comment will be stored in the comment library and will appear in your list of frequently used comments, visible in your comment toolbar. Comments for multiple versions of a question will be labelled as This version and Other versions. Comments from all versions will be available to use on any version, and duplicate comments across versions will be linked to preserve bulk-editing.

1. Optional: If you are an Instructor or Facilitator and wish to share your comments with the grading team, see the next section, Sharing a comment with your team.

Sharing a comment with your team (Instructors and Facilitators)

Instructors and Facilitators can choose to share comments with the grading team. When a comment is shared, it will appear in the comment libraries and comment suggestions of all Graders, Facilitators, and Instructors on the team. The comment is read-only to Graders.

Also see: Using a rubric

1. Select the comment you wish to share.
2. Click the Share with team toggle above the comment content. The comment will be shared.

An evaluation is the collection of annotations, comments, and score left on a question by a grader. When viewing student work, you’ll see a list of evaluations in the right sidebar.

• If nobody has graded the question, you’ll see Your name (new) indicating that you’re looking at your own blank evaluation that has not yet been saved. Once you add an annotation, comment, or score, your evaluation will be saved.
• If another grader has graded the question, you will see their evaluation first. You will also see an Add evaluation button, and if you’re an Instructor or Facilitator, a Clone/Edit evaluation button.

Primary evaluations

Students will only see one grader’s evaluation when their grades are returned. This evaluation is called the “primary” evaluation. While grading, the primary evaluation is indicated by the checkmark icon to the right of the grader’s name. By default, the first evaluation left on the question is marked as primary. Instructors and Facilitators can override this by adding their own evaluation or by selecting another evaluation to be primary.

Cloning evaluations left by others

Instructors and Facilitators have the option to clone another grader’s evaluation. This is useful in situations where an Instructor or Facilitator is reviewing a grader’s work and needs to make changes before the evaluation is shown to the student. Cloning leaves the grader’s work untouched while the Instructor or Facilitator makes changes to their own copy, ensuring that ownership of each piece of feedback is maintained.

Video transcript

Crowdmark makes grading remote exams easy and seamless! This video will show you the remote exam grading workflow.

On your Assessment screen, you’ll see submissisions organized by question. This allows graders to grade individual questions for consistency. You can also have individual graders grading different questions simultaneously. You’re able to see which questions have been submitted, and how many have been graded. Multiple Choice questions are automatically graded. For Image/PDF responses and Text entry responses, click into a question to begin grading.

Once you’re on the grading screen, you’ll see toolbars on the left and right hand side of your screen. On an Image/PDF response, you can use the pencil to hand-draw, place checks, “x”s or question marks, select an area to highlight, or change the colour of your entry by clicking on the droplet icon. Use the garbage can icon to delete anything you’ve added to the page.

Leave comments in your students’ work by clicking on the speech bubble icon. Every time you create a comment, it is automatically saved — creating your Comment library. Comments support markdown and LaTeX, images, or links to additional resources. If you choose to assign point values to a comment, points are automatically applied or removed as you grade. You can pre-populate a question’s Comment library using a CSV file, making it visible to all graders. This allows you to use the Comment library as a rubric. When students receive their grades, they will see all the feedback on their work.

On the right, you can input points using the built in keypad, or the numbers on your keyboard. You can Tag evaluations, and Filter assessments by question, team member, tag or point value. These tools help you get an overview of your grading, and can be a great way to keep track of trends. You can use the Keyboard shortcuts to move through more quickly. You can also view grades and comments from other graders, move to the next ungraded question, and click to move between student booklets. You can also use the Overview Grid to click through to a specific student or question.

Scroll up or down to move through the assessment you’re currently working on, and click on the question labels to grade specific questions.

When grading a Text entry question, your grading tools are designed for text. Use the cursor to select a block to be highlighted or struck-out. Each of these options allows you to attach a comment. Click the icon with the comment symbol to minimize the boxes as you work. Text responses also include the option for students to place links, or downloadable files within their responses.

You’re also able to view and override grading on your multiple choice questions.

Click the name of your assessment to exit the grading screen, then click on the Results tab to see students grades. Here you can distribute the grades to the students, or export the grades to a CSV file. If you’re using one of our LMS integrations, you may also have the option to export grades directly to the LMS.

For more information, check out the Crowdmark Help Centre, or get in touch with Crowdmark Support. We’re always happy to hear from you! Thanks for watching!

To start grading, click a question on your assessment’s Dashboard (Q1, for example). You will see the first student’s Q1. If grading has already started, you’ll be taken to a student’s Q1 that has not been graded.

The grading screen shows the current student’s pages arranged vertically. You can scroll to see all of the student’s responses. On the right is the grading sidebar, where you can see the booklet number and question you are currently grading.

• To go to the next Q1 that has not been graded: Click the Next ungraded button at the bottom of the grading sidebar or type Enter on your keyboard as a shortcut. Please note that “Next ungraded” won’t always navigate to the next student in sequence, especially if others are grading. It is designed to skip booklets that other graders are looking at to avoid collisions.

• To navigate between student papers: Use the arrows beside the Booklet # title or the keyboard shortcuts: l or shift + → to go to the next student, and h or shift + ← to go the previous student. You can also click the Booklet # title to go to a specific booklet.

• To navigate between questions: Click on the blue or green question label (Q2 [Not graded], for example) on the student’s page or use a keyboard shortcut: j or shift + ↓ to go to the next question, and k or shift + ↑ to go to the previous question. You can also go back to the assessment’s Dashboard and select a different question.

You can see an overview of grading by clicking the Overview grid button on your assessment’s Dashboard. Clicking on a grid tile will take you directly to that student’s response.

Instructors and Facilitators can view evaluations with specific tags or a combination of tags, a score or range of scores, or graded by a specific Grader by using the filter tool. You can also use the filter tool on a specific comment using the comment library (see Using the comment library).

Applying filters

1. 1. While grading, click Filter evaluations in the top right corner.
   2. You can filter by question version, team member, tag or score. Numbers beside the filter checkboxes indicate how many evaluations match that filter.

1. Click Apply.

If multiple filters are selected, only evaluations matching ALL filters will be shown.

Navigating filtered evaluations

A bar will appear on the bottom of the page showing which filters you have selected. The grading sidebar navigation will change to allow you to cycle through only evaluations with the selected filter(s). The Next filtered button loops through all pages that match the active filters. Clicking the Back button in your browser will take you to the last page you viewed.

A Printer-Uploader is a team member who prints blank assessments and/or performs the scanning and uploading of pages from an administered assessment. This can be a person at a print shop, a staff person in the department, or another instructional assistant. Printer-Uploaders are allowed to download blank assessments and upload scanned images to the assessment on Crowdmark. They do not have access to grading interfaces, results, or lists of students and team members.

Instructors and Facilitators always have permission to download and upload assessments.

Inviting a Printer-Uploader

Printer-Uploaders can be invited to the course the same way as any other team member (see Managing team members in a course).

QR code errors in assessment upload

While uploading your exams, you may encounter an upload or processing error message.

Here are some common errors and solutions:

• Unprocessable file: Crowdmark could not identify a QR code on the page. It may have been tampered with, cut off, or otherwise obscured. If the text beside the QR code is still readable, you can fix this manually by clicking the Fix button. See “Fixing errors” below.
• The exam image must be a JPG or PDF: Crowdmark currently supports JPG and PDF file formats. If your files are in an unsupported format, you will have to rescan the images or convert them into the proper format.
• The exam must have a density of 150 DPI or more: The quality of your scans is not high enough. The system is calibrated for 200 DPI, so scans with less or more than this setting may appear blurred.
• QR code must be in the right place: Errors can happen if pages are scanned without proper borders, ratios or formatting. Extra white space around the page will cause the QR code to be in a different location than where it can be read. 

Fixing errors

1. Click the All toggle button next to History limit the list to only show Errors.

2. Click the Fix button beside the error message. A modal window will appear.

3. Locate the UUID near the top of the image, beside the QR code. You may have to scroll sideways to find it. This is a combination of letters and numbers that uniquely identifies the assessment.

   

4. Click Select or type in the UUID and begin typing the UUID. Once you’ve located a match in the list, click on it to select it.

5. Click Save. Click outside of the modal window to return to the Uploads list.

After you have uploaded your template, Crowdmark will ask you to define the location and other information about each question. Note that you will be able to make changes anytime after the assessment has been set up.

For each question you will need to specify the:

• Label: This is generally the question number and should be consistent with the labeling on the template. It can be up to 8 characters long.
• Max. points: This is what the question is out of. While grading, graders may enter any non-negative score, but will see a warning if the score is over the max. points specified (entering bonus points are possible).
• Location: By default, Crowdmark places one question label (along with its max. points) at the top left of each page (for example, Q1 [10]). These labels indicate where each question is located and this is where students will see their scores for each question.

Defining questions during assessment setup

By default, Crowdmark will place one question label (with max. points) at the top left of each page as an example. To define your questions:

1. Add question labels to correspond to each question and sub-question in your template by clicking the Add question button. New question labels will appear on your template.
2. Drag and drop the question labels to the locations of the questions on your template. The Q1 label should be dragged to the location of the first question, for example. The question label and score will appear in the same location during grading so it is helpful to place these directly on top of your question content (or in the side margins of the page) so that it will not cover any student work. Question labels define the location where annotations will be available for grading this question. The gradeable area will continue to where the next question label is placed, even across page breaks.
3. Rename the questions to match the naming in your template using the Label fields on the right side of the screen.
4. Specify the max. points (what the questions are out of) for each question in the Max. points fields.
5. If you want to delete a question, click the red “x” beside the Label field of the question.

You should not label multiple choice questions here. They will be automatically labelled and graded on the bubble sheets. The answer key and points for multiple choice questions can be customized after you have finished setting up your assessment. See Using multiple choice questions for more information.

Defining questions from the Dashboard

1. From your assessment’s Dashboard, click Questions in the sidebar. A list of questions will appear.
2. Click the Edit questions button in the upper right corner. You will see the same screen as in the setup.
3. Here, you can move question labels, rename them, change their max. points, and delete questions. Note that if you have started grading a question you will not be able to delete that particular question until all scores and feedback have been removed.

Once you have more than one assessment graded, you can download a list of course grades. The downloaded CSV file can be opened as a spreadsheet and edited in a program like Excel or Google Sheets, or imported to your LMS’ gradebook. The course grades CSV includes colums for:

• Name
• Email address
• Student ID
• Grades by question

• Grades by assessment

You can also export grades for an individual assessment. See Exporting a list of assessment grades for more information.

1. Navigate to your course and click Course students in the sidebar.

   

2. Click the Export course grades button in the upper right corner. Once the file has finished generating, the button text will say Download CSV. Click the button again to download the grades.

   

   The exported CSV file will include a column for each question of each assessment as well as a column with the total scores for each assessment.

   

See also

• Exporting a list of assessment grades
• Downloading student work

Crowdmark offers multiple assessment types, divided into two basic categories: Administered paper-based assessments and Assigned online assessments. Choosing an assessment type depends on whether you would like to administer and collect paper assessments in person or have students complete and submit their work online. Details below will help you choose which workflow will be best for your needs.

Administered

Crowdmark offers three different paper-based Administered workflows:

QR coded Crowdmark booklets are the original, trusted paper-based assessment workflow. Booklets are formatted in Crowdmark, printed, administered in-person, then scanned and graded. Because of their unique nature, QR coded booklets are highly secure, organized and dependable. This is the best workflow to use if you are new to paper-based Crowdmark assessments. 

Non-QR coded Crowdmark booklets offer a workflow for users who would like to take advantage of Crowdmark functions like Auto-matching and Multiple choice bubble sheets, but may require more flexible page formatting. Non-QR coded booklets are also photocopiable, and have the option for extra pages to be added after printing. 

Ready to grade assessments are generally assessments that have not been formatted in Crowdmark, that students may have already completed. They are often worksheets, labs or simple quizzes. This workflow also offers the option for single-page assessments. 

See the following table for a quick comparison of features and requirements between different Administered assessments:

QR-coded assessments are an efficient and easy-to-organize workflow for in-person testing. We’ve compiled this simplified checklist of tips, tricks and best practices for instructors using QR-coded assessments to make sure all your settings are complete for the best possible experience. If you are unfamiliar with QR-coded assessments and need more detailed instructions on how to create one, please see the Administered Assessments section of our Help Center.

Assessment setup

Template

• Has a cover page that contains no information required in the grading interface and space for auto-matching, if required
• Has adequate space or extra pages for student scratch work
• Is set up for double-sided printing

Click here for more on Assessment templates.

Creation

• Double-sided printing is selected to ensure QR codes on every possible page. If a student writes on a page with no QR code, that page will not be available for grading
• Student IDs in the course are numerical only (if using auto-matching). Alphanumeric IDs cannot be read accurately by auto-match technology and significantly reduce the match rate. If this is not the case, see this help video for instructions on replacing existing student IDs
• MCQ bubble sheet(s) are selected (if required)
• QR codes are clear and unobscured by text on the page
• All pages are present and accounted for

Defining questions

• Question labels are placed on each question. Applicable annotation area will extend across page breaks to the next label
• Question points and totals are accurate
• MCQ questions are NOT labeled (if applicable)

Generating booklets

• Assessment roster is up to date. One booklet will be generated for each student in the current list
• Required number of extra booklets are added. Booklets cannot be photocopied because each QR code on each page is unique. Generating extra booklets will make sure you have spares
• PDF of all booklets is generated and stored in an accessible location. Note the file can be re-downloaded from your Administration panel at any time

Multiple choice answer key

• MCQ answer key is filled out
• MCQ settings have been adjusted. For more on default point settings, see #3 in this doc.

Printing Your institution may have specific requirements for printing. For more details please contact your local administrative office.

• Necessary team members have been invited. See this doc for more details on inviting a printer to your team
• Collated printing and/or subset stapling are enabled. If you’re unsure whether your printer has these settings available, we suggest checking in with your local admin office, contacting manufacturer support, or looking up your printer model number online
• Double-sided printing is enabled
• Printed area occupies the full boundaries of the printed page. Any reduction in size ratio will affect QR code legibility
• Printed booklets have strong, full blacks with clearly-visible QR codes and MCQ bubbles

Congratulations, you’re ready for students to write the assessment!

Scanning, uploading, and matching

Scanning Your institution may have specific requirements for scanning. For more details please contact your local administrative office.

• Cutting staples from booklets must not affect QR codes or MCQ bubble sheet content
• Scanning settings are correct according to this help doc. If scan settings are not correct, QR code, MCQ and matching errors will occur
• Pages are scanned in colour or greyscale
• Scans have strong, full blacks with no white space. White space or ‘noise’ in an area that’s supposed to be black will cause QR code and MCQ page errors. If scans are too light, a ‘darker’ scan setting is recommended
• Student work and auto-match details (if applicable) are clearly legible
• Scans are limited to batches under 2000 pages
• Scanning settings are consistent across batches

Uploading Your institution may have specific requirements for uploading. For more details please contact your local administrative office.

• Necessary team members have been invited. See this doc for more details on inviting an uploader to your team
• All scanned batches are present and accounted for
• QR code errors have been fixed. If you have errors that are unable to be fixed, or unexpected upload results, please contact support@crowdmark.com including a link to the assessment you’re working on, email address of the account(s) having trouble, and any relevant PDF files
• Scans from different assessments or versions have routed to the proper place. If uploads are not appearing in the relevant assessment and all uploaders are confirmed to have the correct permissions, please contact support@crowdmark.com including a link to the assessment you’re working on, email address of the account(s) having trouble, and any relevant PDF files

Matching

• Student roster has been synced to ensure all relevant students are available to match
• Student information is complete in the course. Students will not be able to be matched if the required information is not available. See this doc for more details on automated matching, and how to adjust student info
• Run automated matching has been clicked (if applicable)
• Un-matched booklets have been matched manually
•  Extra or blank booklets have been labelled

Congratulations, you’re ready for grading!