Computer Adaptive Testing Promotes Mastery Learning
Principal Supervisors

Professor Shekhar M. Kumta
(Department of Orthopaedics and Traumatology)
Duration

2 years and 5 months
Approved Budget

HK $500,000
 
  • Abstract
  • Brief write-up

Abstract

  1. Mastery of knowledge in certain subject domains is obligatory to the safe practice of medicine. Traditional examination methods however, do not test for Mastery.
  2. Computer Adaptive testing is a sophisticated way of penetrating a student’s knowledge, based on his or her response to prior questions. It provides a precise estimation of knowledge, requiring a few number, but higher quality of questions.
  3. E-Learning resources and Computer Adaptive Testing may support the development of Mastery in some important areas relevant to medical practice as an intern.
  4. This project aims to develop and apply these novel ideas so as to improve the knowledge capabilities of our students prior to their internship.
  5. As intern performance is independently monitored for skills and competencies, we shall be able to evaluate whether our project has a positive impact on capabilities.
  6. This project is also likely to benefit teachers as they will learn a sophisticated method of assessment and will be required to develop high quality questions to assess their students.
  7. It is expected that this project has wider applications and may be extended to any domain in tertiary education.

Brief write-up

Project objectives

Key objectives of the project address the needs of all Clinical Year students (n= 560), including
  1. Develop eLearning Resources focused on Mastery Learning through schematic pattern recognition training – in each of the 4 key subject domains.
  2. Develop a CAT Platform – incorporating a multidimensional variant of computer adaptive testing – for use in 4 selected key subject domains within the Medical Faculty (in each of which mastery is an important requirement for patient care).
  3. Develop Computer Adaptive Tests (CAT) to support Mastery Learning and provide Formative Feedback to student learners.

Activities, process and outcomes

  1. A Computer Adaptive Test (CAT) APP and content management system (CAT Platform) for managing the questions to be used in the APP, have been developed. and tested within the architecture of the APP.
  2. A backbone of the Adaptive Testing Platform is an assessment Item-Bank, in which Item Psychometric data is used as an index of difficulty. We have successfully linked the APP to the IDEAL database (a multi-institutional collaborative Item-Bank). Every use of the questions contributes to their psychometric evaluation and the bank thus improves the precision with which “Mastery” of a subject can be defined.
  3. The APP must be used as an assessment resource that will enable students to work towards the development of mastery. We have developed a workflow that should be an integral part of the educational teaching strategy towards attainment of mastery in various disciplines

Deliverables and evaluation

  1. The Mastery Learning App is made available on the Apps appear on Android and Apple Store.
  2. A Pilot study on the use of the Mastery algorithm in a small cohort of students (n=120) was deemed successful. Only 10% of the students were able to attain Mastery (as defined by the algorithm). This correlated very well with their summative examination assessment scores (R=0.83).
  3. The CATS Platform is being integrated with the IDEAL Assessment Bank Program. This will enable Multi-institutional utilization of the APP and enhance its use across various disciplines, particularly as a Mastery Learning Tool.
  4. Individual Teachers may also use the APP for their teaching and learning activities.

Dissemination, diffusion, impact and sharing of good practices

  1. A Website for the project is established at the following link
    http://www.ome.cuhk.edu.hk/elearnshowcase/profshekharkumta/TDLEG4170502/
  2. The development has been discussed and presented at local regional and international educational workshops (Mastery Learning Using an APPS based Learning Tool: Maharashtra Health Sciences University Workshop, Mumbai, 26 January 2019.; CAT as Mastery Learning – University of Notre Dame, Medical Faculty, Sydney, 3-5 April 2019.; Towards Mastery Learning : Presentation at CUHK Department of O&T Teachers Retreat. 11 May 2019.
  3. The Project will also be disseminated through a workshop at the IDEAL Summit to be held in Vienna, 24-28 August. This will allow access to 30 Institutional members and will greatly enhance its utility but also help us improve its functionality.