Inclusion and Understanding: Assessment and Quantification of Mathematics Exam Problem Characteristics

Inclusion and Understanding: Assessment and Quantification of Mathematics Exam Problem Characteristics

Grant:
Gilbert Whitaker Fund for the Improvement of Teaching
Academic Year:
2017 - 2018 (June 1, 2017 through May 31, 2018)
Funding Requested:
$10,000.00
Project Dates:
-
Applicant(s):
glarose
elande
hbennett
carreonf
paulkess
Overview of the Project:
The primary goal of this project is to understand and develop measures of how accessible exam problems are to different students in introductory mathematics courses, especially to underrepresented groups in mathematics and STEM courses. Additional goals are to determine measures by which course coordinators can quantify the difficulty of exam problems and exams as a whole, and to better understand how to present past exam problems so that students will learn more when using them as a study aid. In sum, these will allow course coordinators in the Mathematics Introductory Program to improve the inclusiveness of their courses, write exams that are more consistent in difficulty and learning objective, and improve student's learning.

We will accomplish these goals by analyzing existing data about student performance on past exams. We expect to be able to isolate a relatively small set of such characteristics that are correlated with significantly worse performance by certain student groups, and heuristic measures that will allow coordinators to understand when problems are likely to be less accessible to these students. We will describe the difficulty of exam problems by determining measures to quantify that difficulty, which we expect to include cognitive demand, problem presentation, and the type of work required of students to successfully solve the problem. Finally, we will use the insights gained from the work on the project to improve the presentation and supporting information students have when using old exam problems to study, with the goal of improving student learning overall.
Final Report Fields
Project Objectives:

The primary goal of this project was to understand and develop measures of how accessible exam problems are to different students in introductory mathematics courses, especially to underrepresented groups in mathematics and STEM courses. Additional goals were to determine measures by which course coordinators can quantify the difficulty of exam problems and exams as a whole, and to better understand how to present past exam problems so that students will learn more when using them as a study aid.

Project Achievements:

The primary outcomes resulting from this project are: (a) a set of standards for mathematics Introductory Program course coordinators providing guidelines for writing more accessible and inclusive exams (attached below); (b) an ongoing recommendation for our coordinators to have a specific exam reader who is looking primarily at issues of complexity and accessibility; and (c) a contribution of momentum and background information for an ongoing and much larger initiative in the Department to improve the assessment in our Introductory Program courses so that it is more supportive of all students and more likely to promote the success and persistence of students from groups underrepresented in STEM.

These largely address the goals of the project. In the standards we have articulated five specific measures by which coordinators may evaluate their exams and how inclusive they are, and the recommendation of an exam reader focusing on this provides a concrete mechanism by which we can facilitate their use of the standards. In addition, an additional and unanticipated achievement of the project is our current and ongoing work through FCI, with support of the NINI grant program and University facilities, to improve our support of instructors' inclusive instruction and to move our Introductory Program courses, starting with math 105, to a mastery learning structure.

Continuation:
The use of the standards developed in the project, and the work through FCI on math 105 and (starting this summer) on math 115 is ongoing.
Dissemination:
We are distributing the standards document to new and continuing course coordinators each semester. This reaches nine coordinators each year (including five graduate student co-coordinators), and indirectly informs the perspectives of all instructors of the affected courses (math 105, 115, and 116, especially). The number of instructors in these courses is significant; in the academic 2019-20 year, there were 141 different instructors teaching them. We have also provided the results of our work to our Associate Chair for Education, and Director of Undergraduate Programs.
Advice to your Colleagues:
Our primary advice is perhaps not entirely portable: we were fortunate to work with a group of very dedicated and capable colleagues, and to have very significant support from the College and from CRLT. We recommend this to all seeking to address these types of difficult pedagogical questions.
PRINTER VERSION
SEND BY EMAIL