Designing assessments that assure learning in the age of AI is a challenge. How can we assess learning and be sure that students have achieved the learning outcomes we say they have?
Interactive Oral assessments (IOs) gained attention during the COVID-19 lockdowns as an alternative to on-campus exams (Logan et al., 2020) and is now considered a viable, scalable and sustainable means of assuring learning in the age of GenAI (Lodge et al., 2023). An IO is a form of authentic assessment that can promote the development of employability skills, enhance student engagement, and address academic integrity.
In this post, we share some of our early learnings from a project to design, implement and evaluate IOs at The University of Sydney Business School – let’s talk!
How is an IO different from other types of oral assessment?
The IO is not an exam (like a viva voce) where students are asked a set of questions. In the IO, the marker uses prompts to guide a conversation with the student. Through the conversation, the student has the opportunity to demonstrate relevant knowledge and skills in real-time. While viva voces have a reputation for inducing stress, students tend to have more positive experiences of IOs due to the inclusion of preparatory in-class activities, a familiarity with free-flowing conversation and the relevance of this type of assessment to the real world (Logan et al., 2017).
The IO is intended to replicate an authentic setting such as a workplace environment or scenario. This gives students an opportunity to act in the role of their ‘future selves’, to the extent that many students choose to dress professionally for the role even when not required. To further replicate the authentic setting, many IOs allow students to bring notes or resources into the session to support the conversation.
IOs can be facilitated either online or on-campus in any discipline in which real-world scenarios can be imagined (for examples, see Karltun & Karlton, 2014; Logan & Sotiriadou, 2020; Ward et al., 2023). They are viewed as scalable, with IOs having been run in courses of up to 800 students (Logan et al., 2023).
About the project
Our year-long project, supported by a Strategic Education Grant from The University of Sydney, aims to pilot the use of IOs in both small and large courses. The project team, a group of educational developers, is working closely with Business School coordinators to design, implement and evaluate IOs across a number of different courses.
Project lift off
Our project launched in March 2024 with a series of events run by two experts in IOs, Danielle Logan-Fleming and Popi Sotiriadou from Griffith University. This included a presentation to the Business School about the key features of IOs, and dedicated workshops for our project participants and other interested colleagues on how to design and facilitate IOs. Following this, Danielle and Popi ran some university-wide workshops which were well attended. The Educational Innovation team have shared their learnings from the day on the Teaching@Sydney blog.
In these events we were fortunate to be joined by Ju Li Ng, an academic in the Business School who shared her experience of running similar forms of oral assessment in business education, and two of her students who provided some valuable insights and feedback on their experience of participation in IOs.
Phase 1: Starting small
This semester we are piloting IOs in 5 different courses across the School in the areas of digital transformation, global management and culture, management and organisations, and building organisational customer experiences. We’re starting small, with units of <60 before testing IOs in larger courses.
Guided by the 9 design stages shown above, we worked with course coordinators to co-design IO assessment tasks. The early design phase involved linking the IO with a previous assessment task (for example, by asking students to extend or synthesise their previous learning), designing a scaffolding activity to prepare students for their IO, and developing assessment criteria, rubrics, and conversation prompts. After the implementation of each IO, we will be running students focus groups and an interview with the coordinator and markers.
Examples of real-world scenarios
Example 1: an interview scenario
In a course about digital transformation coordinated by Danny Gozman, the teachers acted in the role of a partner in a management consultant firm specialising in digital transformation. Students were asked to act in the role of an interviewee going for a job as a partner in the firm. The scenario required students to reflect deeply on their previous experience of developing and managing a digital transformation project (their previous assessment) using a specific reflective framework. In the IO, the student was invited to discuss their reflections of team processes and outcomes, make recommendations for developing and managing future digital transformation projects, integrate examples and evidence, and demonstrate a professional demeanour throughout the interview.
Example 2: giving advice to a CEO
In a course about management and organisations coordinated by Elly Meredith, students are put in the role of a management consultant giving advice to the CEO of a company related to some aspect of workplace management that is impacting adversely on productivity. In this scenario, the management consultant is given exactly 5 minutes to present their solution and then has 10 minutes to engage in an unscripted conversation with the CEO about the advice. The requirement is that the student, while focusing on one aspect, needs to explain and justify these actions on other sections of the business. So, if they are looking at culture within one part of an organisation, they will also need to focus on how this will impact the whole structure of the company. As such they need to examine all sections of the business and the role that managers and organisations play.
Example 3: sharing leadership approaches with a senior manager
In a course about global management and culture coordinated by Echo Liao, students play the role of a junior manager, and are asked by their senior manager (who wants to step back from a project) what they would do if they were to take over the leadership role of an international team which has a distinct set of challenges.
Our key learnings so far
While we are only at the start of our journey, there are a few things we’ve learnt about how to effectively design and facilitate IOs:
- Designing and embedding scaffolding tasks to prepare students for their IO is essential. In two pilot units, students reported this helped them feel comfortable with doing the ‘real’ IO.
- The IO is effective when it extends or synthesises students’ learning from a previous task
- Careful alignment between the IO conversation prompts and rubric support clear and efficient marking processes
- IOs can be run effectively either face-to-face or online (e.g. on Zoom)
- Unsolicited feedback from students suggested they view this type of assessment positively and value its relevance to the real-world
- Assessors find IOs engaging to facilitate and the live marking easier than expected
- It is important for markers to schedule in some short breaks if assessing a high number of students in a day
Phase 2: Scaling up
In Semester 2 we will be carrying our learning forward, and piloting IOs in three medium-sized courses (100-150 students). We are expecting that the shift in context from small to larger courses will require greater attention to aspects such as providing effective hands-on facilitation training for the marking team, and setting up robust processes to support logistics at scale such as scheduling IOs.
Building our community
During our project launch we invited colleagues to join an IO Community of Practice (CoP) to share and access ideas, task designs and rubrics developed as part of our pilot study. In addition to sharing resources via Teams, we plan to coordinate some informal sharing events for the CoP over the coming months.
If you would like to find out more about our project please get in touch! stephanie.wilson@sydney.edu.au
Acknowledgments
We’d like to acknowledge Associate Professor Popi Sotiriadou and Danielle Logan-Fleming (Griffith University) for their support, and for facilitating IO design workshops at the University of Sydney Business School in March 2024; colleagues and students who have participated in Phase one of the project including Danny Gozman, Echo Liao, Elly Meredith, and Mike Seymour; and Ju Li Ng for presenting her experience with Meraiah Foley and Vanessa Loh of implementing oral assessments; and Danny Liu and the team from Educational Innovation at The University of Sydney for their continued support of the project.
Feature image: Christina@wocintechchat.com, Unsplash.
About the author
A lifelong learner and educator, Alison is an education developer lecturer with the Business Co-Design hub at the University of Sydney's School of Business. She has a PhD in Theoretical Chemistry, and is a qualified Science and Maths high school teacher.
Jo Nash is an Educational Designer with Business CoDesign at the University of Sydney. She brings a practical and strategic approach to educational design with extensive experience in unit coordination, lecturing, marketing, advertising and strategy. She is particularly interested in the codesign of units to help teaching outcomes, plus enhancing academic integrity at the same time as student learning outcomes.
Educational Developer, Business Co-Design
Stephanie is a Senior Lecturer and Deputy Director (CLaS) with the Business Co-design team at Sydney University and Senior Fellow of the Higher Education Academy (SFHEA). She enjoys working with others to explore new approaches to learning and teaching inspired by design practice and the arts.