Priorities

1. Create efficiency and quality improvement across assessment pathways

Removal of the Clinical Examination Retest

The AMC clinical retest examination was no longer offered to IMGs from January 2019. The retest was historically offered to IMGs who were assessed as having a borderline performance as an opportunity to provide the AMC with a further indication of the IMG’s ability. However, there were often long periods between the main clinical exam and the retest, resulting in the retest exam becoming equivalent to a main exam but with half the number of stations.

An IMG’s performance in the clinical examination is now scored as either a pass or fail overall. In conjunction with this change, examination performance feedback will be reviewed to provide more useful information to IMGs. This is expected to be introduced in 2019/20.

Remote Marking of AMC Clinical Examinations

The remote marking initiative was first conceived when the NTC was built in 2013 and will enable the marking of AMC Clinical Examinations without requiring an examiner to be present in the examination room. The ability to assess clinical examinations using video for the AMC has been substantiated through the AMC Clinical Examination Appeals process, and the technology is well developed by the NTC Team and technology partners.

The project completed initial phases of technical, operational and strategic development in June 2019 with piloting to take place in July/August 2019.

The implementation of this initiative is aimed to improve the quality of AMC assessment.

AMC & Medical Deans of Australia and New Zealand (MDANZ) Standard Setting Exercise (SSE)

In May 2019 the AMC undertook a standard setting exercise (SSE) of the MCQ examination.

The AMC MCQ examination is set at the standard of a graduating Australian medical student about to enter into the first day of internship. While there is no common exit examination for Australian medical graduates, this project allows each university to administer different MCQ items that are all calibrated to a common difficulty scale using item response theory and the AMC MCQ question bank.

The SSE was constructed using a modified Angoff and other standard setting approaches and involved a 1.5 day program of work. The exercise was facilitated by a psychometrician and educational expert working with the Chair of the AMC CAT MCQ Development Group as co-facilitator. Participants included members of the MDANZ group and AMC discipline leaders along with the AMC executive, assessment management and operational teams.

Examiner Decision Making Project

In February 2019 the AMC initiated a research project to better understand how examiners form judgements with the aim improving the quality of the AMC clinical examination and adding to international assessment literature. The project, which involves researchers reviewing video footage of AMC examiners marking clinical examinations, was released as a public tender with Flinders/Otago Universities the successful respondents to the tender.

2. Facilitate knowledge sharing within the industry by initiating partner and stakeholder workshops

AMC Assessment Summit

The AMC Assessment Summit, held on 21 October 2019, was an initiative to bring together a broad stakeholder representation group, including MBA/AHPRA, Directors of Medical Training, IMGs, international assessment affiliates and experts, along with AMC Directors to discuss the AMC assessment pathways.

The four key areas of focus for the Summit were to:

1. Inform the future strategic direction of non-specialist IMG assessment practice, including identifying priorities for future work
2. Share information on current approaches, including new initiatives in assessment by the AMC and both national and international partner organisations
3. Identify common challenges and opportunities for improvement in non-specialist IMG assessment, and
4. Strengthen relationships between key stakeholders to promote and support best practice in non-specialist IMG assessment.

Some of the considerations highlighted in the Summit were:

• the importance of patient centric care as a priority to achieve fair, valid and reliable assessment
• support of the continued work on the clinical and WBA assessment pathways
• the requirement for better engagement with IMGs
• the requirement for improved feedback to IMGs in order to inform self-directed learning and support, and
• greater need for understanding longitudinal outcomes of assessment through research and work with MBA and AHPRA.

Initial feedback from attendees suggested strong support for a 3 step pathway which incorporates an MCQ examination, clinical examination and WBA.

The Summit Facilitator is currently preparing a report to document key outcomes of the summit and inform future strategy for AMC assessment pathways. Key areas will include identifying priorities for future work by the AMC in collaboration with key partners. The document will be a key component of a broader and continuing body of work developing the AMC non-specialist IMG Assessment Strategy.

AMC National Workshop on Workplace-Based Assessment

The AMC held a National Workshop on Workplace Based Assessment in Melbourne on 17 September 2018 – see Workplace Based Assessment continues to strengthen article in highlights.

3. Increase collaboration to optimise use and potential of the NTC

NTC Business Planning

The National Test Centre (NTC) is a purpose-built and state of the art facility designed for high-stakes examination delivery. In the 2018/2019 year, the AMC NTC was used for examination delivery by seven external clients, including various specialist colleges and universities.

With the NTC operational for six years, the AMC has commenced reviewing the NTC business plan and opportunities for sustainable growth, including ways to improve the client service model, diversify the client base, and seek future expansion in alignment with the AMC Strategic Plan.

NTC Website

A dedicated website for the NTC was launched in March 2019 to raise the profile of the NTC as a leading enterprise in medical assessment and increase opportunities for research and development, collaboration and other partnerships.

The site was developed in conjunction with the broader marketing and business strategy for the NTC, in particular engagement with new and existing clients such as specialist colleges, universities, other standards bodies and technology partners.

UPGRADES TO TECHNOLOGY OF THE NTC

Camera Upgrades: The cameras inside the exam rooms were upgraded to enable piloting and implementation of the remote marking project. The project has increased camera angles from two to four within each exam room and improved video image quality. At the same time a solution that automatically segments recorded video footage on a per candidate basis was implemented.

Observation Room Workstation Upgrades: The workstation PCs in the observation room were upgraded to improve the overall user experience for viewing the exam room video and bookmarking tool.

Improved Digital Signage: Additional signage screens were been installed around the facility along with a with a content management system to improve wayfinding and branding for the AMC and External Client examinations.

4. Update the delivery of AMC publications and handbooks

AMC App

The AMC currently publishes a Handbook of Multiple-Choice Questions to assist IMGs in preparing to take the AMC MCQ examination. The AMC App is a project aimed to digitise the publication into an interactive mobile application.

The app will utilise archived MCQ questions to create a study resource for IMGs and medical students preparing for examinations. A pilot product has been developed and feedback provided by focus groups of IMGs and Australian medical students to guide the next stages of development.

Anthology of Medical Conditions

The AMC Anthology of Medical Conditions was first printed 2003 and is one of four major publications recommended to IMGs as an essential study preparation tool which forms the basis of content for the MCQ and clinical examinations.

An editorial group was established to review the publication with a focus on ensuring the content reflected considerations for at-risk groups within the Australian health community, population health, and Aboriginal, Torres Strait Islander and Māori health. The editorial group includes Aboriginal and Torres Strait Islander representation and the project progress and objectives were presented to the AMC Aboriginal, Torres Strait Islander and Māori Strategy Group for consultation and feedback. The newly constituted AMC Aboriginal and Torres Strait Islander Standing Committee, when formed, will continue to advise on the direction of the project.

The Anthology group has commenced the re-write of the publication which will improve the legal, ethical, and organisational (LEO) content along with an increased focus on Aboriginal and Torres Strait Islander health in line with the AMC strategic plan and pillar.

Content is being finalised for 134 Presentations, and 63 LEO and Aboriginal and Torres Strait Islander topics, as well as an additional chapter providing an overview of medical practice in Australia.

As at 30 June 2019 the content had undergone drafting and internal review, and feedback provided from workshops consisting of IMGs and Australian medical students.