Overview
Standardisation and calibration can be used in addition to moderation to seek to ensure fair and consistent assessment outcomes across a module and/or programme. These activities take place before marking is undertaken rather than after as in the case of moderation.
Standardisation or “parity marking” is module specific and can be used when more than one academic staff member is involved in the assessment of a module e.g. a project module. It involves discussing the requirements of an assessment (the brief, learning outcomes and marking criteria) and then individually marking a small sample of assessments. Marks are then compared, discussed and agreed. The aim of standardisation is to ensure that consistent and fair grading judgements are made for all students taking a specific module.
Calibration is programme/subject rather than module specific and involves the discussion of different assessments from a range of modules at different levels in order to reach a shared agreement of grading judgements. The aim of calibration is to allow assessors from the same subject area to make consistent and fair judgements which are stable over time on different pieces of work across a programme.
Calibration can also involve academics from across a number of institutions in order to seek to ensure consistency of awards across the sector. External examiners may also be involved in such act.
Key Principles for Standardisation and Calibration
- Developmental - Standardisation and calibration activities are opportunities to share good practice and learn from others and develop a shared understanding.
- Collaborative - Assessment documentation and grading judgments need to be discussed in a collaborative, non-judgemental and good humoured way.
- Planned - Activities should be planned to ensure that all assessors are able to participate and there is sufficient time allocated for dialogue.
- Well Managed - See 'Managing Standardisation and Calibration Meetings' section below.
Good Practice
Holding a face-to-face meeting(s) before the start of a delivery period using previously completed assessment work (without annotations) can enable staff to interact develop a shared understanding of grading judgements. Meetings also enable any issues with the clarity of assessment briefs to be identified and allows staff to provide more consistent assessment support to students.
Where assessment briefs differ to previous years e.g. examinations, case-studies, a timetable of standardisation meetings should be set before the start of a delivery period which takes into account assessment deadlines in order to should be drawn up to ensure standardisation activities do not unnecessarily delay feedback opportunities.
Involving new academics in standardisation activities even if teaching on different modules will help to shape their judgements in their other modules.
To allow standardisation and calibration of samples of oral assessments and practical exams to take place, it is beneficial to record assessment outcomes.
Managing Standardisation and Calibration Meetings
Sift and select a small sample of work in advance. Ensure assessments are not already annotated/pre-graded.
Stage 1:
Explain the importance of standardisation/calibration and the developmental nature of the activity – not only for new staff but for all group members.
Stage 2:
Ask the group to examine the assessment brief and the requirements therein, as well as the General Assessment Criteria for that level and any mark scheme. Based on the evidence provided, identify any issues/inconsistences in the assessment documentation. Discuss and agree the baseline requirements for a pass, as well as any higher categories.
Stage 3:
Ask the group to provide assessment feedback and a provisional mark for each assessment in the sample using the relevant assessment proforma. This can be done in the participant’s own time or during the meeting.
Stage 4:
Ask each member to submit their assessment proformas anonymously to the Chair/convener.
Stage 5:
Discuss discrepancies and the rationale for these. Allow all members to present their viewpoints and ask questions. Focus on evidence and not opinions.
Stage 6:
Determine a consensus judgement for the work with a clear rationale for this. Repeat Stages 5 and 6 for further assessments (It is likely that initial judgements will change as a result of discussions on earlier assessments) The following video shows how a large-scale calibration activity can be managed effectively:
Advance HE (2018) Case-study: Calibration: the Australian Model [online], available from https://www.youtube.com/watch?v=OjRp0WXoB-A
Further Reading
Advance HE (2018), Calibration Standards: What, Why and How? [Online] available from: https://www.advance-he.ac.uk/sites/default/files/2019-12/Calibration%20synthesis%20report.pdf
Bloxham, S., Hudson, J., Outer, B. and Price, M. (2015) External peer review of assessment: an effective approach to verifying standards? Higher Education Research & Development, 34(6), pp 1069-1082.
Royce Sadler, D. (2013) Assuring academic achievement standards: from moderation to calibration. Assessment in Education: Principles, Policy & Practice, 20(1), pp 5-19.
Royce Sadler, D. (2014) The futility of attempting to codify academic achievement standards. Higher Education, 67, pp273–288
Sefcik, L.T., Bedford, S.B., Czech, P. and Smith, J. (2017) Embedding external referencing of standards into higher education: collaborative relationships are the key. Assessment & Evaluation in Higher Education, 43(1), pp 45-57
Watty, K., Freeman, M., Howieson, B., Hancock, P., O’Connell, B., De Lange, P. and Abraham, A. (2014) Social moderation, assessment and assuring standards for accounting graduates. Assessment & Evaluation in Higher Education, 39(4), pp 461-478.