How to design the best Assessments in the Era of AI?

State of affairs

One of Oxford University Press’s first premises regarding the use of technology and the optimism that it usually triggers in its novel forms should be that education should drive technology, not the other way around. AI technologies have been worrying the whole educational establishment through the past years since cheating in assessments has become more accessible and quicker than ever, and the usual pedagogical system of memorizing and repeating materials is becoming obsolete in the academic context.

According to the numbers facilitated by Oxford University Press, “most teachers think digital resources—including AI-powered technology—have had ‘significant’ or ‘some’ positive impact on educational outcomes for their students (63% of school teachers, 59% of English language teachers). ” The most significant impacts so far are on content preparation, assessment, and personalisation. Only 23% of UK teachers felt prepared, compared to more than half of those outside the UK. [1]

However, not all is dark regarding the use of AI in education. Though approaches to AI in education differ considerably between countries, Oxford University Press’s research shows that “both English language and school teachers are optimistic—but cautious.” [2]

Furthermore, some universities do not consider all kinds of use of artificial intelligence (AI) in assessment as plagiarism. In the University of Glasgow's Code of Student Conduct, it is claimed that we should reconsider how we design assessments to ensure that students can demonstrate skills beyond purely knowledge recall. They add that the use of AI could be accepted for debugging code, experimenting, or playing with ideas, for instance, or dealing with content that would be online; the use is considered legitimate. The AI-generated content should be just informational and lack analytic or critical intentions, which the student should implement. [3] So, where is that limit, and how can it be confronted when it’s crossed?

Many say these days that performance should be based on soft skills rather than mechanically absorbing technical knowledge. For instance, substacker Bechem Ayuk talks about freedom and quotes Paulo Freire and his concept of praxis [4] in relation to building new forms of assessments adapted to new technological possibilities. Bayuk paraphrases Freire, claiming that education “should be liberating and empower learners to challenge oppressive systems. ”

He summarizes his theories by proposing that “students should be active, not passive, in their journey and learning process. The interconnectedness between academic and social tissues needs to be understood at an early stage. Participating actively in the learning process is not only encouraging, but it also helps students learn more than they perceive. ”

Another technique proposed by Freire is the active construction of knowledge, which is known as constructivism in psychology. In this technique, knowledge must be actively applied to construct meaning, collaborate with peers, and reflect on one's experiences.
[5]

Ultimately, not only students are using AI for their educational journey. A whole market of tools for using AI to create personalised and more efficient assessments is rising. Sean McMinn, Director of HKUST | Center for Education Innovation, describes the subject of identifying students’ use of AI technologies as complex and case-specific since “a lot depends on the intended learning outcomes and the pedagogical approach used. ” [6] In his article, McMinn distinguishes between student-generated content and student/AI-generated content. Regarding student-generated content, the assessment design should restrict the use of AI, and for student/AI-generated content, the assessment would imply the implementation of AI technologies for the evaluation.

Performance-based assessments

As the introduction mentions, Bechem Ayuk advocates exploring assessment methods that are different from conventional formats. These methods, such as performance-based assessments, have been proven susceptible to AI-powered cheating since soft skills and reasoning drive them. PBA is based on the learning process rather than on an outcome. This kind of assessment should encourage the students to apply their knowledge and skills in practical, real-life contexts, carrying out authentic tasks.

This method departs from the soft skills expected to succeed in the “real” world and assesses the steps, strategies, and decision-making processes students employ to reach the final product, following Paulo Fraire’s concept of “praxis, ” which emphasises integrating theory and practice. Freire believed that education should be liberating and empower learners to challenge oppressive systems. Freire’s concept is similar to John Dewey's “learning by doing ” theory, which encourages students to make meaningful connections between ideas and real-life applications.

In 2019, Havard published a study showing that students learn more than they think when actively participating in the learning process. Moreover, PBA promotes metacognitive processes, as students must reflect on their performance, identify areas for improvement, and develop strategies to enhance their future performances.

Criteria for Performance-based Assessments

To implement PBA, Bechem Ayuk proposes some questions to be asked before implementing PBA: [7] The main criteria is a broader alignment in terms of goals, making clear what the goal of the learning is so that the student can engage with the goal rather than with the methodology. The context in which these assessments should be applied must be meaningful for the student. Unlike traditional methods, the outcome is not evaluated; instead, the process and performance, quality of work and impact. Ultimately, the criteria need to be fair, objective and transparent.

Assessment Approach

There are two ways of approaching assessments regarding the use of AI. One is trying to prevent its use, and the other is encouraging it. For each case, McMinn discloses a list of conditions essential to consider when implementing AI in assessments.

When trying to prevent the use of AI

In this case, McKinn encourages face-to-face, closed-book assessments. The ideal context format would be project-based or problem-based, such as community projects. Another idea, in this case, is to present the work orally after the research, including reflections, which can also be provided on a paper. As mentioned earlier, the tasks must be complex enough to represent real-world situations so intuition, decision-making, and reasoning can be applied in addition to knowledge. These tasks must be investigated over a sustained period so that a notable evolution exists. Lastly, the assessments can be incorporated into class, which means they should be carried out.
When encouraging the use of AI

If students can use AI in their assessments, McMinn recommends some parameters to implement this implementation to create adequate evaluations. These parameters can take the shape of tools like text, image, or code-generative AI tools. These generative tools can be used in different stages, such as the planning stages of the assessment task. McKinn points out that AI can also be used as a core part of the task, for self-testing, or as a copyediting tool.

Implementating AI in Assessments

Before implementing the tools, UCL University of London encourages in its dossier, a discussion about integrity in the use of AI for assessments with their students, as well as considering ethical issues, ensuring accessibility, providing support, and setting transparent and fair evaluation criteria, ” – claims McMinn. [8] Such a discussion should encourage trust and responsibility in the classroom. The responsibility of the teaching force is also to provide guidelines on fundamental principles such as transparency, critical thinking, feedback, and AI literacy, which are essential for successfully integrating AI tools into education.

Process-Oriented AI-Responsive Classroom Practices

Nick Potkalitsky, who teaches 9th graders with an optimistic approach to AI, advocates for a process-based education rather than education as an end product. In his article “Rediscovering the Power of Process”, he posits that language should be a process for discovery, and writing should be seen as a critical tool for thinking and exploration rather than a way to convey pre-formed ideas. Such a process requires knowledge scrutiny and emotional engagement; the process is essential and shouldn’t be overshadowed by a final product. The focus on high-stakes grading leads to AI tool misuse, such as plagiarism, one symptom that the learning process should be more important than the outcome. One way could be implementing process papers that allow students to reflect on their AI usage, “shifting the focus from efficiency to utility-driven technological engagement.” Furthermore, traditional research papers can be replaced by these kinds of initiatives.

His next Substack entry, “Quantify or Qualify? The Future of Assessment in the Age of AI, ” [9] shows a survey of the standardisation of academic assessments. He goes through behaviourist theories of Edward L. Thorndike and B.F. Skinner. The first focused his research on the “measurable aspects of education, emphasizing reinforcement and the quantification of learning outcomes,” whereas the latter innovated in the field of teaching machines and programmed instruction. These principles “reinforced learning through immediate feedback, pioneering early forms of what would evolve into computer-assisted learning. ”

Departing from these two premises, he explains the possibilities of AI-supported classroom assessments, which can be used to personalise the learning experience. He looks at the historical trajectory of education, which tends towards “measurement, quantification, and product orientation.” Thanks to this quantification, it can be better assessed how to reshape education into “a streamlined, manoeuvrable discipline capable of assessing and anticipating the impacts of nuanced changes within both local and broader educational contexts. ”

For a while, the education sector has teamed up with an implementation of digital devices to turn into a product-based discipline. Yet, AI systems' advancements are not comparable to the mere use of devices. The devices can help expand the possibilities in the reach of resources. Yet, AI technologies can help generate adapted materials based on the needs and development pace of the students, making learning processes way more efficient. The system aligns better “with the real processes. ”

Potkalitsky advocates for a system that better aligns with the natural processes we and our students engage in daily throughout the academic year, including the rise and advancement of AI systems. He has created a process-oriented AI Responsive Classroom practice composed of the following steps.

Empowering Student Agency
Dedicated Project Time
Personalized Instruction
Process-Focused Assessment:
Revision Opportunities
Reflective Writing Choices
Flexible AI Integration

Goals and Conclusion

The dossier published by Oxford University Press proudly confirms that they and other UK Russell Group of universities have already produced their principles on ethical usage to guide their members and educators in general. ” [10] Ultimately, McMinn’s proposition to discuss and consider the role of AI in the educational journey can help to define the overarching goals, which is one central point in the successful implementation of AI technologies in academic assessments. [11] Wayne Press, Global Product Director, Education Division, OUP, confirms that “being clear on the learning goals and the role that technology can play in supporting them is crucial. ” [12] But, the most important question, before any implementation of AI technologies in assessments, and independently from whether there are restrictions with the use of AI or not, “teachers should ask whether the use of AI tools significantly changes the student experience and whether they are making a difference to the efficacy of teaching to ensure that these chances are managed effectively. ” [13] – says Professor Victoria Nash, Director of the Oxford Internet Institute.