The work of the Quality Assurance of Examinations Group

The work of the Quality Assurance of Examinations Group

Challenges and options in assessment Workshop to plan an international approach to the assessment of postgraduate training in Occupational medicine Barcelona November 2006 02/08/20 Prof Lesley Southgate. [email protected] What must be done to establish a credible assessment programme that can encompass different approaches. 02/08/20 Prof Lesley Southgate. [email protected] A student was a sked to

multiply 389 by 34. His answer is shown below: 389 34 x 1556 11470 13026 02/08/20 Working individually, give this student a mark out of 10 Prof Lesley Southgate. [email protected] What is a blueprint Today it is a map and a specification for an assessment programme It may be a simple content matrix But it can also include further specifications, for

example test/assessment methods It supports content validity by ensuring sampling and coverage It prevents overlap and the inclusion of hobby horses 02/08/20 Prof Lesley Southgate. [email protected] Validity: How has the content been determined 02/08/20 Does the assessment actually sample the domain that is being assessed? Has a written blueprint been derived before the assessment was planned and implemented?

The content should show evidence of referencing to external documents and data What is the method of sampling from it? Prof Lesley Southgate. [email protected] The common and important conditions form one dimension of a matrix the other is the categories of Good Medical Practice 02/08/20 Prof Lesley Southgate. [email protected] Initial Blueprinting for the UK Professional Linguistic Assessment Board (PLAB) Test

02/08/20 Prof Lesley Southgate. [email protected] External review PLAB 200 Written EMQ OSCE Knowledge Theoretical diagnosis Management 02/08/20 Prof Lesley Southgate. [email protected] Content

Blueprint Blueprint 02/08/20 Prof Lesley Southgate. [email protected] Developing the blueprint Performance standard A first year Senior House Officer What does a first year SHO see? 02/08/20 Prof Lesley Southgate. [email protected] What does a first year SHO see? Medical school core cases Workload

surveys A&E depts 02/08/20 Common & Important Questionnaires to recent Prof Lesley Southgate. [email protected] candidates Hospital discharge data 130 presentations presentations 130

02/08/20 Prof Lesley Southgate. [email protected] Attributes Attributes 02/08/20 Prof Lesley Southgate. [email protected] 02/08/20 Prof Lesley Southgate. [email protected] Selecting the examination modality XX XX Prof Lesley Southgate. [email protected]

XXX X 02/08/20 OSCE OSCE Written Written Appraisal Appraisal Item writing - OSCE Experienced OSCE writers Proposed stations Piloted

Editing group Added to bank 02/08/20 Prof Lesley Southgate. [email protected] Provisionally Accepted Item writing- Extended Matching Questions Item writing workshops Introduction to the task Question template 02/08/20

Prof Lesley Southgate. [email protected] Authors: Code: Knowledge/skill area (blueprint column heading): Presentation/diagnosis (blueprint row heading): Theme: ____________________________________________________________ (eg State both skill and content, eg diagnosis of headache) Options A K B L

C M D N E O F P G Q H

R I S J T Instructions: For each _______________________ below, choose the SINGLE most likely _________________________from the above list of options. Each option may be used once, more than once, or not at all. Item 1: Authors: Code: Knowledge/skill area (blueprint column heading): Presentation/diagnosis (blueprint row

heading): 02/08/20 Prof Lesley Southgate. [email protected] Authors: Code: Knowledge/skill area (blueprint column heading): Presentation/diagnosis (blueprint row heading): Theme: ____________________________________________________________ (eg State both skill and content, eg diagnosis of headache) Options A K

B L C M D N E O F P G

Q H R I S J T Instructions: For each _______________________ below, choose the SINGLE most likely _________________________from the above list of options. Each option may be used once, more than once, or not at all. Item 1: Theme: ____________________________________________________________

(eg State both skill and content, eg diagnosis of headache) 02/08/20 Prof Lesley Southgate. [email protected] Authors: Code: Knowledge/skill area (blueprint column heading): Presentation/diagnosis (blueprint row heading): Theme: ____________________________________________________________ (eg State both skill and content, eg diagnosis of headache) Options A K

B L C M D N E O F P G

Q H R I S J T Instructions: For each _______________________ below, choose the SINGLE most likely _________________________from the above list of options. Each option may be used once, more than once, or not at all. Item 1: Options A

K B L C M D N E O F P

02/08/20 Prof Lesley Southgate. [email protected] Instructions: For each _______________________ below, choose the SINGLE most likely _________________________from the above list of options. Each option may be used once, more than once, or not at all. Item 1: Authors: Code: Answer: Knowledge/skill area (blueprint column heading): (Write in full) Presentation/diagnosis (blueprint row

heading): Theme: ____________________________________________________________ (eg State both skill and content, eg diagnosis of headache) Options A K B L C M D N

E O F P G Q H R I S J

T Instructions: For each _______________________ below, choose the SINGLE most likely _________________________from the above list of options. Each option may be used once, more than once, or not at all. Item 1: 02/08/20 Prof Lesley Southgate. [email protected] Item writing- Extended Matching Questions Item writing workshops Introduction to the task Question template Peer review Formal editing Piloting

02/08/20 Prof Lesley Southgate. [email protected] Piloting Test administered to volunteers Under-performs returned to authors Revision Editing group Rejected 02/08/20 Prof Lesley Southgate. [email protected] Accepted into

bank Results Results of of initial initial pilot pilot Stage Stage N N Mean Mean score score Medical Medical student student PRHO

PRHO 21 21 62.66 62.66 36 36 70.28 70.28 SHO SHO 20 20 73.90 73.90

02/08/20 Prof Lesley Southgate. [email protected] Work programme Matrix accepted Question bank being developed Piloting in progress Test construction agreed 02/08/20 Prof Lesley Southgate. [email protected] Test construction 17 4 categories

Diagnosis 02/08/20 Management Investigation Prof Lesley Southgate. [email protected] Others Test construction Disorders of childhood 133 Disorders of the elderly Cardiovascular system

16 categories 02/08/20 Peri-operative management Prof Lesley Southgate. [email protected] Work for today Similar process starts by deriving the matrix 02/08/20 Prof Lesley Southgate. [email protected] The common and important conditions/problems

form one dimension of a matrix the other is the categories of Good Medical Practice or domains which map to them 02/08/20 Prof Lesley Southgate. [email protected] Test methods as part of the blueprint Test methods for assessment programmes can only be selected once the purpose and content are established

02/08/20 What is the best trade-off between fidelity and efficiency. What is the programme for writing and refining test material? How are marking schedules developed and reviewed where relevant? Is new test material piloted before it is used? Prof Lesley Southgate. [email protected] Is the assessment reliable Tests are likely to be unreliable if they are poorly constructed or too short Examiner bias Report Coefficient alpha and/or KR-20 But they provide information about the

reproducibility of marks, not pass/fail decisions 02/08/20 Prof Lesley Southgate. [email protected] Standard setting It is only possible to set a sensible standard if the purpose of the examination is clearly set out. The most important thing about a standard is who sets it 02/08/20 Prof Lesley Southgate. [email protected] Standard setting If

the standard is criterion referenced how and by whom, have the internal rules within and between individual components been set. How many, and to what standard, of the criteria are mandatory? Is compensation allowed? 02/08/20 Prof Lesley Southgate. [email protected] Standard setting Are these rules subject to change if there is an unexpected outcome and if so who has the authority to direct such a variation and how and where is it documented? How has the pass mark been derived for largescale tests of competence What recognition is given to measurement error in arriving at the final outcome?

02/08/20 What is the reproducibility of the pass/fail standard? Prof Lesley Southgate. [email protected] How are examiners selected and trained. What mechanisms are in place to monitor examiner performance And to ask unsatisfactory examiners to leave the panel? Do examiners undergo equal opportunity training before they participate? 02/08/20 Prof Lesley Southgate. [email protected]

Feedback to candidates One of the most important aspects of any assessment programme is the effect it has on learning and preparation for individuals and the profession. How is feedback given to individuals and groups about the outcomes of an assessment? Finally the mechanisms for appeal about the outcome should be documented and reported. 02/08/20 Prof Lesley Southgate. [email protected]

Recently Viewed Presentations

  • Cybersecurity Primer Information Security Awareness Importance of Cybersecurity

    Cybersecurity Primer Information Security Awareness Importance of Cybersecurity

    Cybersecurity is Safety. Security: We must protect our computers and data in the same way that we secure the doors to our homes. Safety: We must behave in ways that protect us against risks and threats that come with technology....
  • DNA Necklace Lab - twhspalmer.weebly.com

    DNA Necklace Lab - twhspalmer.weebly.com

    DNA Necklace Lab Materials clear sports drink cup 15 mL disposable test tube test tube rack microcentrifuge tube transfer pipette ice cold isopropyl, 70% ethanol, or rubbing alcohol dish soap solution Prepare Make sure you know which tube is your...
  • 1.04 Employ product mix strategies to meet customer expectations

    1.04 Employ product mix strategies to meet customer expectations

    PRODUCT SERVICE MANAGEMENT. Concepts and procedures necessary to obtain, develop, maintain, and improve a product or service mix in response to market opportunities.. PRODUCT: The . goods and services . a business will offer to its . customers
  • UNICORE in XSEDE: The Journey Down the Road

    UNICORE in XSEDE: The Journey Down the Road

    XSEDE Vision. XSEDE: enhances the productivity of scientists and engineers by providing them with new and innovative capabilities. and thus. facilitates scientific discovery while enabling transformational science/engineering and innovative educational programs
  • Broadband Service Switzerland - ACN Inc.

    Broadband Service Switzerland - ACN Inc.

    ACN Line Rental Product Training Portugal Contents Opportunity & Benefits Service Offering Customer Sign-Up Process Important Information Provisioning Process Opportunity & Benefits Population: 10.6 million Fixed lines in use: 4.22 million Benefits Increase in customers with a bundled line rental...
  • NUR 696 Evidence Based Practice Sarah Koepp, Marla

    NUR 696 Evidence Based Practice Sarah Koepp, Marla

    Judy Blauwet, DNP, MPH, BSN, RN former Chief Nursing Officer at Avera Mckennan Hospital and University Center . Lori Popkes, MBA, BA, RN, NE-BC current Chief Nursing Officer at Avera McKennan Hospital and University Center
  • Low Back Pains Missing Piece Diagnosing the Sacroiliac

    Low Back Pains Missing Piece Diagnosing the Sacroiliac

    Lumbar spine. Anatomy - Spine. 24 vertebrae. Base of Skull to Pelvis. Building blocks. Discs between vertebrae. Cushions between bones. ... Design: Triangle vs. Round. More surface area. Unique coating allows for bony ingrowth. Ingrowth creates fusion. Permanent solution, 4X...
  • Kiwanis Club of Jacksonville Beaches Florida District ...

    Kiwanis Club of Jacksonville Beaches Florida District ...

    Ross Edwards To give primacy to the human & spiritual rather than the material values of life. To encourage the daily living of the Golden Rule in all human relationships. To promote the adoption & the application of higher social,...