13 Research methods, statistics, critical review and evidence-based practice13.7 Qualitative research
13.7 Qualitative methods in instrument development
2 min read
Written by Resident Doctor Committee · Published
Types of validity
Type | Question it asks | Detail |
|---|---|---|
Internal validity | Does the study support its causal claim? | Reverse causation threatens it: exposure and outcome are recorded together, so the direction of effect cannot be fixed |
External validity | Does the result generalise? | A benefit shown in a selected sample may not transfer to other populations |
Content validity | Does the scale cover the whole domain of the construct? | Qualitative work with the target group, through interviews or focus groups, is a standard way to establish it |
Face validity | Does the measure look, at a glance, as if it measures what it claims? | Judged by those completing it; the most superficial form, resting on no statistical test |
Poor face validity lowers engagement and completion.
Content validity differs from face validity in depth: content validity asks whether the domain is fully covered, while face validity asks only whether the measure appears right.
Content validity in scale development
Content validity covers relevance, comprehensiveness and comprehensibility for the intended construct, population and use.
High internal consistency does not show that every domain is covered, because a scale can measure one narrow feature over and over.
Generating items
Concept elicitation explores the experiences of patients and clinicians before the items are fixed.
Interviews give depth and privacy, focus groups develop ideas through interaction, and observation reveals practices that never appear in accounts.
Items should use participants' own words, so asking about "loss of pleasure" reaches more people than asking about "anhedonia".
Cognitive interviewing
Cognitive interviewing examines how people understand and answer an item. Tourangeau's model has four stages: comprehension, retrieval, judgement and response selection.
Think aloud asks respondents to describe their reasoning, while probing asks targeted questions about meaning or why an answer was chosen.
Studies run several small rounds of 5 to 10 participants, revising and retesting between rounds.
Handing over to quantitative testing
Quantitative work then examines reliability, internal structure, floor and ceiling effects and responsiveness.
Item reduction should not strip out rare but important content, such as suicidal ideation, merely because it correlates weakly with the rest.