Psychometrics

​

-

​​

What are the Psychometric concepts (e.g., reliability, validity, bias review, item analysis). Align it with Cambridge Assessment Network's philosophy. A101: Introducing the Principles of Assessment - A102: Introducing Assessment Practice - A103: Assessment Data and Statistics - A104: Psychometrics in Educational Assessment? 

​

-

Psychometric concepts like reliability, validity, bias review, and item analysis are the technical toolkit Cambridge uses to make assessments fair, meaningful, and usable in real classrooms and systems.​

 

Cambridge assessment philosophy

Cambridge Assessment Network tends to frame quality through a small set of core principles: validity, reliability, fairness (including freedom from bias), standards/comparability, and practicality/impact. These principles run across A101–A104, with A101 focusing on the big ideas, A102 on classroom and operational practice, A103 on data/statistics, and A104 on deeper psychometrics.​

• A101 emphasizes that all technical decisions (items, mark schemes, grading) must ultimately serve clear purposes and valid interpretations of scores.​

• A102 brings those principles into marking, moderation, and everyday assessment design for teachers and practitioners.​

• A103 and A104 extend the same principles into statistical evidence: item statistics, reliability estimates, and models like Classical Test Theory and Item Response Theory.​

 

Validity

Cambridge treats validity as the central question: “Does this assessment support the interpretations and decisions we want to make?” rather than just “Does the test look suitable?”.​

• Valid assessments:

  • Measure the construct they are supposed to measure and are based on what learners have actually been taught.​

  • Minimize construct‑irrelevant variance (language load, cultural references, formatting or scoring quirks that affect scores but are not part of the intended skill).​

• In the A101–A104 logic, validity is:

  • A design issue (alignment with curriculum and learning outcomes).​

  • A fairness issue (avoiding bias and inappropriate inferences about learners).​

  • A statistical issue (using data from A103/A104 to check that item and test behaviour supports the intended construct and decisions).​

 

Reliability (including marking)

Reliability is about consistency: similar learners should receive similar scores if nothing important has changed.​

• Cambridge discusses reliability through:

  • Stability of scores across occasions and forms (would a learner’s result be similar at another time or on an equivalent paper?).​

  • Consistency of marking, especially where human judgement is involved.​

• A102 highlights practical levers for reliability:

  • Clear, well‑designed mark schemes and tasks that reduce ambiguity in marking and response.​

  • Marker training, standardisation, and monitoring, with acceptance that some task types trade a bit of reliability for richer evidence.​​

• A104 extends this into psychometric indices:

  • Classical reliability coefficients (e.g. internal consistency) and marking reliability analyses.​

  • Use of models (CTT/IRT) to understand how items contribute to overall score precision.​

 

Fairness, bias review, and accessibility

Fairness is treated as a core principle on par with validity and reliability, not an optional add‑on.​

• Cambridge’s accessibility guidance stresses:

  • Appropriate linguistic demand, clear wording, and removal of redundant or confusing information.​

  • Avoidance of cultural bias and stereotypes so that scores reflect the construct, not background knowledge linked to culture, gender, or socioeconomic status.​

• Bias review, in this philosophy, is both:

  • Qualitative: expert review of content, language, context, images, and mark schemes for potential disadvantage to groups.​

  • Quantitative: using item‑level data (A103/A104) to spot items that function differently for subgroups even at the same overall ability (e.g. differential performance patterns signalling possible bias).​

• Fairness feeds directly back into validity: an assessment cannot really be valid if scores are systematically distorted by irrelevant factors like culture, language variety, or disability.​

 

Item analysis and psychometrics

In Cambridge’s framing, psychometrics is the statistical machinery that checks and documents whether the principles above are actually being achieved in practice.​

• Item analysis (A103/A104) typically examines:

  • Difficulty: how many candidates answer correctly or achieve high marks, used to judge whether an item is appropriately targeted.​

  • Discrimination: how well an item distinguishes higher‑performing from lower‑performing learners, using statistics such as item‑total correlations in CTT.​

  • Distractor behaviour: whether incorrect options are plausible and attract mainly lower‑performing students, signalling good functioning rather than confusion or mis‑keying.​

• Two main psychometric approaches are highlighted:

  • Classical Test Theory (CTT), working with test scores, item difficulty, and discrimination indices to support reliability and basic equating.​

  • Item Response Theory (IRT), using models that estimate item parameters and learner ability independently, supporting more advanced uses like equating, adaptive testing, and item banking.​

• In the Cambridge philosophy, item analysis is not just technical:

  • It provides evidence for validity (items behaving in line with the construct), reliability (contributing to consistent scores), and fairness (flagging potentially biased items).​

  • It creates a feedback loop from data back to test design, mark schemes, and classroom practice, which aligns with how A102–A104 position “assessment literacy” for educators and assessment professionals.​

If you want, the next step can be to translate these into ESL‑specific examples (e.g. how you would review a reading comprehension item or a writing rubric for validity, reliability, and bias in your own context).

​

-

​