Assessment Hub

Scale development

Scale development should be theory-driven, iterative, and focused on construct validity rather than simply producing a reliable measure.

Key recommendations for future scale development research:

1. Begin with a strong theoretical foundation

  • Clearly define the construct and its boundaries.
  • Specify how it differs from related constructs.
  • Conduct a comprehensive literature review before generating items.
  • Consider where the construct sits within broader theoretical or hierarchical models. 

2. Generate an overinclusive item pool

  • Create substantially more items than will appear in the final measure.
  • Sample all relevant content domains.
  • Combine deductive methods (literature review, existing measures) with inductive methods (interviews, focus groups, expert consultation).

3. Prioritise item quality

  • Use clear, simple, unambiguous language.
  • Avoid double-barrelled questions, jargon, and highly skewed items.
  • Ensure items are appropriate for the target population.
  • Consider carefully whether reverse-scored items are necessary. 

4. Assess content validity early

  • Obtain feedback from both experts and members of the target population.
  • Conduct pilot testing and cognitive interviewing where possible.
  • Refine items before large-scale data collection. 

5. Use large, heterogeneous samples

  • Recruit representative participants from the intended population.
  • Aim for at least 10 participants per item, with larger samples preferred.
  • Use multiple independent samples where possible. 

6. Evaluate structure using factor analysis

  • Use EFA to explore dimensionality.
  • Use CFA to test and cross-validate the structure.
  • Preferably use separate samples for EFA and CFA.
  • Remove items with weak loadings or problematic cross-loadings. 

7. Focus on unidimensionality and construct representation

  • Do not maximise coefficient alpha at the expense of content coverage.
  • Avoid the attenuation paradox, where highly redundant items increase reliability but reduce validity.
  • Prioritise broad construct representation over extremely high internal consistency. 

8. Establish comprehensive validity evidence

  • Assess:
    • Convergent validity
    • Discriminant validity
    • Criterion validity
    • Predictive validity
    • Incremental validity
  • Include comparison measures during initial scale development rather than waiting until later stages. 

9. Assess multiple forms of reliability

  • Internal consistency alone is insufficient.
  • Consider test-retest reliability, inter-item correlations, and other reliability indicators.
  • Aim for adequate reliability while maintaining conceptual breadth.

10. Use modern psychometric methods where appropriate

  • Consider Item Response Theory (IRT) in addition to classical test theory.
  • Use IRT to optimise item selection and develop short forms.
  • Examine item functioning across the full range of the construct. 

11. Treat scale development as iterative

  • Revise both the measure and the conceptualisation of the construct in response to empirical findings.
  • Cross-validate findings across samples.
  • Continue accumulating evidence for validity after publication. 

 

Summary

Good scales are developed through a rigorous and iterative process involving strong theory, comprehensive item generation, expert and stakeholder input, robust psychometric evaluation, large representative samples, and extensive evidence for construct validity. Reliability is necessary, but construct validity should remain the primary goal. 

 

For more information, please refer to these papers:

Clark & Watson (1995)

Clark & Watson (2019)

Morgado et al. (2017)