The rise of generative AI poses new challenges to educators, especially in terms of assessment design. For now, there is the potential that students could overtly (or covertly) use generative AI systems to synthesise eloquent written responses to assessment questions. With this new challenge, how then do we ensure students are still developing underlying key reasoning skills, rather than simply outsourcing the process to generative AI systems? Additionally, should we be encouraging students to start using generative AI responsibly? And what role would assessment design have in achieving these goals?

Consider my own professional area, for example: law. In the study of law, there is a long-held method for teaching legal reasoning and the process of applying a law to a situation. This method is often condensed to the acronym: ‘IRAC’. In the IRAC method, ‘I’ stands for ‘issue’ as the first step in legal reasoning is to identify a legal issue that needs to be resolved. Next, ‘R’ stands for rule – what is the legal rule (law) that needs to be applied to the scenario under consideration? ‘A’ is for application, where the legal rule is applied to a legal problem or factual scenario to analyse what could be the legal outcome in a particular situation. Finally, ‘C’ is for conclusion where a view is offered as to the potential resolution of the legal issue being investigated.

Prior to the proliferation of generative AI language models, students studying law units would commonly be required to master the IRAC method as part of developing their underlying legal reasoning skills. The importance of this method has often been underscored in assessment tasks. It has been common for students to be provided with a hypothetical factual legal scenario in assessments, in which they are subsequently asked to advise one of the parties in the hypothetical of their legal position. Such a question inherently requires the use of legal reasoning – the IRAC method – to reach a conclusion about the legal position of a party in the hypothetical scenario.

The rise of generative AI language models, however, directly challenges not only this form of assessment but, more importantly, the use of hypothetical legal problem questions as a tool for developing a student’s legal reasoning skill set. For we have entered an age where it is possible for a student to simply copy a factual scenario into these types of generative AI and prompt it to produce a response using the IRAC method advising the party under consideration of their legal position.

The rise of generative AI language models, however, directly challenges not only this form of assessment but, more importantly, the use of hypothetical legal problem questions as a tool for developing a student’s legal reasoning skill set.

In response to this possibility, several questions come to mind. First, how do we ensure students still develop their legal reasoning skills? If we continue to rely upon traditional hypothetical legal problem questions for assessments in the world of generative AI, this runs the risk that students will not ever engage with the IRAC method themselves if they can outsource the process to AI.

At first glance, this may not seem problematic. Would it not be better if students could simply learn to incorporate generative AI into their decision-making process when trying to reach a legal conclusion? In part, yes: it would reflect what they will likely be doing in their professional lives. However, it does not negate the need for students to possess their own legal reasoning ‘tool kit’.

It is widely reported that generative AI systems do not always produce accurate outcomes. Indeed, one lawyer in New York, recently, and somewhat catastrophically, fell foul of this problem. New York lawyer, Steven Schwartz, reportedly used the generative AI system ‘ChatGPT’ to research a brief for court and inadvertently submitted to court ‘hallucinated’, or fabricated, legal precedents.

So, even if students do use generative AI to reach a conclusion to a legal problem, they still need the legal reasoning ‘tool kit’ to be able to then integrate the accuracy and relevancy of the AI produced response to the legal situation under consideration.

This then raises the question of how do we design assessments which encourage the development of the legal reasoning ‘tool kit’ whilst balancing the competing possibility that students could use generative AI to produce a cohesive response? One possibility could be that instead of providing students with a standard hypothetical problem or question to which they produce a response, why not instead provide them with a generative AI produced response to a problem question? Students could then be asked to a) analyse it for the elements of the IRAC method and b) rewrite the response to ensure it accurately provides a legally reasoned conclusion to the problem scenario. Another option for assessment design could be to ask students to generate an AI produced response to a task and then analyse the merits of the produced response.

Even if students were to use generative AI as part of preparing their responses to these tasks, they are still required to assess the accuracy of the generative AI produced material. This requires students to not only have an understanding of their course materials and content but also engage with legal reasoning processes to assess if the AI response adequately ‘meets the brief’. Students who simply rely on generative AI without critically assessing its output are, for now, at least, less likely to perform to the required (or minimum) standard. Such approaches therefore maintain the development of a legal reasoning process, assess key legal principles or concepts required, and encourage students to assess the accuracy of any generative AI produced scenario.

A counter to all this is to simply say: “Ban the use of generative AI outright in assessments!”. There are several problems with this approach. If students are using generative AI to address non-assessed problem questions throughout the semester, rather than practising the skill set for themselves, they are still likely to be underdeveloped in their reasoning skills come time for an AI-prohibited assessment.

Perhaps more importantly, if we ban the use of generative AI outright, when are students going to learn how to use generative AI appropriately? As I have alluded to above, it is more than likely that upon entering the workforce graduates will need to know how to use generative AI appropriately and ethically as it relates to their chosen profession. Indeed, Lexis Nexis, the producer of one of the leading legal databases, recently announced the launch of their own generative AI system, informed by Lexis Nexis’ proprietary database. This seems likely to be a game-changer for those in the legal profession and those engaged in legal research.

Is it not important, then, to begin encouraging students to use generative AI responsibly? A key part of the problem for the New York lawyer, mentioned above, who reportedly failed to use generative AI responsibly, is that they seemingly did not understand the limitations of generative AI, nor how to appropriately use it to achieve their aims.

Should we not be designing assessments that, together with assessing knowledge of key concepts and essential reasoning ‘tool kits’, also encourage students to ethically engage with the use of generative AI in a way that is meaningful to their field of study and later, their chosen profession?

The rise of generative AI calls into question traditional assessment design. It challenges educators to consider new ways to ensure students develop their underlying intellectual ‘tool kits’ in the face of technology which can, albeit with varying successes, mimic such reasoning processes. Additionally, it prompts educators to consider if we should not only be seeking to develop subject specific skills but also instil responsible generative AI principles.