By Thao Vu, Zachari Swiecki and Ari Seligmann
Posted Tue 12 March, 2024

With regular reports of the latest AI advances and how the newest tools and platforms promise amazing feats, it is hard to keep up and to find a clear path through the hype, hysteria and ensuing storms. To assist, a dedicated group of colleagues spent the past few months thinking through a range of complex, contemporary issues and formulating several step-by-step guidance documents to provide starting points for everyone to engage with issues and  implications for higher education.

We acknowledge that AI assistance will continue to play increasing roles in our lives. For example, we currently rely on AI assistance to guide us from place to place, to help us search for information and recommend movies, music, reading, recipes, etc., and to help finish sentences in our texts and emails. With the increasing strength of Large Language Models (LLMs), AI can provide even more assistance finding, summarising and comparing texts, transcribing and translating, and using prompts to generate texts, images, videos, sounds, etc. So how do we generate assessments that can provide evidence of learning in an age of expanding generative AI?

The quest to reform, redesign, rethink and reimagine our assessments is upon us. To help everyone navigate the promising journeys we have formulated three sets of documents with prompting questions to help guide directions, and a further guide to additional resources. 

The first guide questions if and how an assessment needs to change. To achieve the learning outcomes required of your unit, do students have to demonstrate individual human knowledge and skills independently? Or can they use a specified range of AI tools to assist them in advancing their knowledge and skills? Put another way, do you need to design AI into the assessment and support the collaborative production of knowledge and skills? Or do you need to design AI out of the assessment and secure the activity to ensure that it can adequately reflect AI-independent human abilities? Which components or subcomponents of assessments need to be demonstrated individually and which can be collaborative efforts? What combinations of individual work and collaborative work should be scaffolded within a unit or across a set of units? The first guide prompts considerations of key issues through scaffolded queries.

After using the first guide to establish orientation, the next two guides help navigate considerations for integrating AI into assessment and for bracketing AI out of assessment, respectively.

Drawing on experiences of co-developing this guidance, and on her expertise in education design, Thao Vu proposes three fundamental (old friend) principles for educators when navigating the myriad possibilities associated with assessments and AI.

We need to be deliberate, not reactive, in deciding and justifying how we align our assessment design with the changing technological, pedagogical and knowledge contexts.

Firstly, we need to be deliberate, not reactive, in deciding and justifying how we align our assessment design with the changing technological, pedagogical and knowledge contexts. In other words, we must be able to justify in what ways the assessment is (a) relevant to the AI-infused future that our students will engage with, (b) facilitative of students’ development of required knowledge, skills and values, and (c) capable of yielding valid evidence of student learning. The prompting questions in the three guides can assist everyone with this decision making process. 

Educators can – and should – engage with others in co-designing solutions to the wicked problem of assessment and AI.

Secondly, educators can – and should – engage with others in co-designing solutions to the wicked problem of assessment and AI. For instance, we can use the guides to prompt ongoing conversations with other academic colleagues, students, curriculum leaders, faculty educational designers, industry partners, potential future employers, relevant members of the public, and so on. This co-creation process itself is a mechanism to ensure assessment authenticity and a systematic approach to reforming assessments. 

Thirdly, we can use the need for AI-driven assessment change to also think more programmatically about assessment. How will we develop better coordination of diverse, fit-for-purpose assessment methods that can synthesise rich information about student learning across related sets of units or be woven through entire courses? How can we imagine assessments more longitudinally and use them to provide early diagnoses of students’ performance, and track their progress and holistic achievement over a course of study? How can we balance the proportion of low and high stakes assessments to inform and defend our decisions about student progression? In the rapidly evolving landscape of assessment and AI, the enduring core principles of validity considerations in assessment design, assessment authenticity and programmatic assessment can anchor us while we cultivate innovation and navigate transformations.

Furthermore, Zachari Swiecki suggests that, if you are designing AI into your assessments, consider organising the submission to collect evidence of the learning and doing processes rather than simply relying on the final product. Explicitly evaluating the process in addition to products will make it clearer how the students interacted and collaborated with AI systems and others. Scaffolding assessment design processes will also give you the chance to offer more formative feedback to students along the way. Carefully consider what process evidence you want and need students to produce, what forms/formats you want to collect it in and how the process documentation will factor into the marking regime. Try to stay flexible and find a balance between getting the evidence you need to have faith in the integrity of the assessments and overloading your students and teaching team.  

With these things in mind, please try our guides and worksheets to work through the relevant issues for your teaching context. Please post comments to join in ongoing conversations about AI in assessment. Share your experiences, further guidance, guiding questions, and useful guideposts discovered along the journey.

References

Ajjawi, R., Tai, J., Dollinger, M., Dawson, P., Boud, D., & Bearman, M. (2023). From authentic assessment to authenticity in assessment: broadening perspectives. Assessment & Evaluation in Higher Education, 1-12.

Fawns, T., & Schuwirth, L. (2024). Rethinking the value proposition of assessment at a time of rapid development in generative artificial intelligence. Medical Education, 58(1), 14-16. doi:10.1111/medu.15259

Heeneman, S., de Jong, L. H., Dawson, L. J., Wilkinson, T. J., Ryan, A., Tait, G. R., … & van der Vleuten, C. P. (2021). Ottawa 2020 consensus statement for programmatic assessment–1. Agreement on the principles. Medical teacher, 43(10), 1139-1148. 

Masters, G. (2013). Reforming education assessment: Imperatives, principles and challenges, Australian

Education Review 57, 9-31. http://research.acer.edu.au/aer/12/ 

TEQSA. (2023). Assessment reform for the age of artificial intelligence. https://www.teqsa.gov.au/sites/default/files/2023-09/assessment-reform-age-artificial-intelligence-discussion-paper.pdf