Creating Exams

One of the common misconceptions about inclusive teaching practices is that it excludes the use of tests, which for many instructors and subjects offer an efficient and clear measure of student learning and achievement. Research reveals that this is not the case, in fact, repeated testing continuously proves to be one of the most effective learning methods.

Mapping Exam Questions

The foundational element for creating more equitable exams comes from a detailed understanding of what the exam is testing, ensuring that it is appropriately testing skills and concepts you desire, and communicating transparently what students can expect. To achieve this, it is incredibly valuable to create an exam map. Doing so has long-lasting benefits in terms of being able to re-use and adjust questions easily as well as providing detailed and meaningful feedback.

Mapping process

The goal of an exam map is to align questions with course learning outcomes/objectives and levels of complexity. Depending on the class and instructor, this can be done at multiple levels of complexity. In some classes a whole exam may be associated with a single learning outcome and broken down by a small number of objectives. In other classes there may be 50 small-scale learning objectives spanning a test. Part of this is personal preference, part may also reflect levels of detail in accreditation standards in some disciplines. In the example below you can find a basic approach to creating such a map with the following columns:

• Question Type: This can be useful for tests that have a variety of sections and approaches to align types of questions with appropriate outcomes and cognitive processes (particularly when there is a mix of multiple-choice and open-ended questions).
• Question Text: In some topics you might have specific individual questions and in others you might have parts (names, numbers, locations, etc.) that can be adjusted while still addressing the same outcomes and cognitive processes.
• Course Learning Outcomes/Objectives: We recommend viewing the Learning Outcomes and Objectives section for further guidance on constructing these.
• Cognitive Process: Here we have used examples from the cognitive domain of Bloom’s Taxonomy, but you may also consider other frameworks if they align well with your course. The aim is to recognize how many questions are within a certain cognitive task and whether the task aligns with the task as described in the outcome/objective.

Question Style/Type	Question Text	Course Learning Outcomes/Objective	Cognitive Process
True/False	There was more time between the evolution of the first T. Rex and the first stegosaurus than between the asteroid that led to the extinction of dinosaurs and the first Neanderthals.	Identify the key organisms and dates associated with each era, and respective period, in the geologic timescale.	Understanding (in the class these animals would be identified as key exemplars for their respective eras).
Multiple-choice (with a video clip played for the class)	Based on the clip you will watch, how would you label the following show? a. Techno-culture b. Popular culture c. High culture d. Youth culture	Identify examples and characteristics of different forms of contemporary culture in the United States.	Application (students would not study the show being clipped, but the selected clip would have elements that align with the appropriate answer).

Use the map to be transparent in test preparation

One of the most frustrating questions for many instructors is “will this be on the test?” because it implies that students are interested only in those things that will be on the test and disinterested in what is not on the test. For some students, this perspective comes from experiences in which exams are viewed as adversarial — attempts to trick students. — rather than opportunities to learn. Few instructors intend to use the layout, structure, and/or format of an exam as a means of deception or trickery (an exception being an exam in which the learning objective is to navigate a complicated exam). By mapping exams through learning outcomes and objectives, we can share those outcomes and objectives with students and give them a clear sense of what to expect in terms of approximate percentages or weights for particular objectives. When objectives are clear and well-structured this helps guide students to a meaningful understanding of material and effective study habits that will align with exam goals to measure achievement more accurately.

Use data to understand class performance

The creation of a map also aids instructors after the exam. By examining student success across questions and having those questions aligned with types and outcomes/objectives, instructors gain the ability to see where there may be significant gaps in the delivery and practice of particular content areas and skills, as well as when certain questions may be outliers which could suggest either misunderstanding or varying degrees of difficulty (which may be intentional or not). By examining these data, we can fill in common gaps going forward with current students, as well as plan course revisions for future semesters. This data can be shared at the individual level with students to give them a better understanding of gaps in their knowledge based on specific course objectives. In mastery-oriented courses, where grades are aligned with final performance on a particular objective, this data may be used to replace earlier grades when a student has demonstrated enhanced mastery of knowledge or skills.

Consider how topics and tasks are organized throughout the test

Using your map, consider whether you want questions dealing with the same objectives or outcomes organized together or separately. If students are expected to identify correct approaches to solve wide-ranging problems, then you might want the questions distributed throughout the test. This will ask students to make choices about their approach for each question. If the topics or objectives are more distinct, you may consider grouping all similar questions together and even offering a heading to help students direct their attention and memory appropriately. For example, in an introductory German language class, we might label one section “Verb Conjugations,” another section “Vocabulary,” and a third section “constructing sentences,” to help students focus their attention on the task at hand and potentially help them make use of their time during the exam. When making these decisions, consider the context where students will use these skills in the future. Will it be in a context where they will know what type of problem they are solving? Or will these skills be used in more abstract situations, where identifying the type of problem is also essential? If identifying the type of problem is important, that should be an outcome or objective and students should be trained to do this during the semester (this process of analyzing and/or evaluating a problem is a high-level cognitive task – sometimes much more complicated than the application or understanding necessary to arrive at a solution).

Key Elements for Making Tests More Inclusive

Incorporate testing into the learning process and not just high-stakes end-of-semester assessments

The power of testing, as described in the book, Make It Stick (Brown 2014), comes from its use as a repeated practice tool. When used well, tests can quickly reveal where students have gaps in their knowledge and provide meaningful guidance on what to do next to fill those gaps. This, however, only works well when students have the opportunity to use the test as practice and receive feedback quickly. This is most effective when questions can be graded automatically in Brightspace, drawing from large pools of potential questions. Many mastery-oriented systems extend this idea, allowing students to repeat quizzes and exams until they demonstrate a certain level of proficiency, and that demonstration is what counts toward final grades no matter how many tries a student needs. When questions cannot be in formats that lend themselves to automated grading, students can spread attempts across a semester and those who have already sufficiently demonstrated mastery of particular objectives may be allowed to skip portions of an exam related to that already demonstrated mastery. If done strategically, this can help distribute the grading and feedback burden across the semester.

Use exam wrappers to have students reflect on their learning from the test process

An exam wrapper is an activity where students reflect on their test performance. When receiving the test back, include reflection questions to help students better understand their performance and how they can enhance it for future tests. This is particularly useful in a class with multiple tests, so that students can develop effective study habits. There can be two layers of the exam wrapper. Some include questions that students submit alongside the test, before receiving a score. Common examples are:

• Predict your score on this exam.
• Rate your studying for this exam.
• List specific study habits that you used on this exam.
• What did you find easiest or most challenging on the exam?

More commonly, or alongside the questions above, are questions students answer when receiving the test back with a score. Common example questions are:

• What specific study strategies did you use and how much time did you use for each?
• What was surprising about your experience with the exam?
• What gave you the most trouble (certain types of questions, certain concepts, etc.)?
• What study strategies will you continue and what strategies will you change as you prepare for the next exam in this class?

An example of providing opportunities to demonstrate increased understanding

Be intentional about timing  

Timing is one of the most complicated parts of an exam, particularly in-person exams, which often have very specific timeframes. One question you might begin with is asking yourself whether it is important in the context of the outcomes and objectives that they be completed in a timed manner. If so, consider how much time is appropriate and needed for each question type when creating questions. Be transparent with students about expectations regarding timing to answer question types and provide opportunities for practice under time constraints, with discussions about strategies for being efficient. In many cases time constraints in exams are not tied to the specific objectives. In these cases, one goal in creating the exam should be that time is not viewed as a stressor for students. 

Give students practice in the testing environment  

In many instances the classroom environment is different for a test than on non-test class days.  In some classes, this involves tests taking place in entirely different rooms. In other cases, desks are reorganized or spread further apart. Other times, changes are made to lighting or projecting timers on the screen. Any of these changes can lead to additional stress that may diminish a student’s ability to accurately represent their learning and skills. When possible, having students take a test in the same environment in which they are learning material can enhance comfort and memory. When this is not possible, consider having some type of “practice” session or review session in a structure that will emulate the testing environment.

Grading and Feedback at Scale

Mastery-oriented tests and quizzes can provide immediate feedback and practice

When adopting repeatable testing or quizzing into the learning process through Brightspace, we have the opportunity to provide instant and meaningful feedback. For questions that can be automatically graded (multiple-choice, true/false, etc.) you can automate specific feedback based on a correct or incorrect option. As described above, this is particularly valuable with the development of foundational knowledge, or the core content and ideas that students should know and recall readily, often in the process of using that knowledge with high accuracy levels for engaging in more complex application and analysis tasks. When drawing from a sufficiently large pool of questions, holding high standards and potentially short time restrictions can be a sufficient assessment of mastery in itself. In questions for this purpose that have clear right and wrong answers, automated annotations can offer immediate detailed feedback and explanations of common misconceptions.

Simplifying feedback through Gradescope  

For tests that have more detailed problem solving, feedback can often be made more efficient, while still personal, through systems that facilitate common feedback for common errors. One such system, Gradescope, supports automating this process through allowing instructors to review one problem at a time and drag and drop common feedback chunks. This allows us to offer more depth than when we need to rewrite feedback for everyone. Additionally, by tracking how frequently some errors arise, we can view this as detailed and systematic feedback regarding the challenges students experience in each problem on a test and tailor review and future class iterations to pre-emptively address these common challenges.

Use the test map to identify specific areas for questions requiring feedback

Many instructors express frustration with a perceived need to assign multiple-choice tests due to the time constraints involved in grading other question types that might allow students to express their learning and skills more fully. While tools like Gradescope can help to alter this equation, it is not true in every instance. By using a map of the test that connects questions with objectives and outcomes, you can also identify which particular objectives would benefit most from more feedback or hand-graded types of questions. Even in very large classes of 200 to 500 students it might be feasible to grade one or two questions by hand for every student with the knowledge that those questions serve a specific purpose in demonstrating student learning. 

Next Steps

Use this Excel template to map an exam (you can download your own copy to work on) and answer the questions in this Creating Exams worksheet