Explain the following terms:

🤔 Why measure our mission?

• Teaching to the test
• External validity
• Measurement choices might lead to strategic behaviour

☑️ Types of validity

• Face: Degree to which the measure makes intuitive sense (relying on what you know about the concept)
• Content: Degree to which a measure represents all critical dimensions of a given concept.
• Construct: Degree to which the concept is related to other measures that it is supposed to be related to
• External: Degree to which the measure is generalisable across contexts (e.g., across countries, populations, physical settings).

🤔 What makes a measurement good?

• Validity

• Reliability: the degree to which a measure is consistent (stable and accurate) and reproducible

• Like theory, a measurement is good when it does the job we want it to!

• Bueno de Mesquita and Fowler (2021): when it measures our mission!

📏 Adcock and Collier’s framework

📏 Adcock and Collier’s framework

Common measurement challenges

• Desirability bias
• Hawthorne/experimenter demand effects
• Unrepresentative samples
• Attrition

📋 List experiments

• Imagine a researcher wants to know how many students use ChatGPT in their assignments
• What happens if I just ask?
• Let’s try it…

📋 List experiments

• Many survey respondents answer sensitive questions in a misleading way because they fear social (or other) consequences of admitting something.
• But we can use list experiments to measure the % people use ChatGPT without requiring any one individual to disclose whether they do.
• We randomly divide our sample into two lists—one has one option more than the other.
• We then estimate the ATE of being given the option.

📋 List experiments

📋 List experiments

• Thinking back to last week’s topic—what is the fundamental problem of causal inference?
  • We cannot know how many an individual would have chosen had they been in the other group.
  • We cannot know at the individual level whether they would have chosen the option of interest.
• What’s our solution to not being able to compare individuals?
  • Comparing the mean of our two groups!

📋 List experiments

You can also access the survey by clicking on this link or pasting it into your browser: https://lse.eu.qualtrics.com/jfe/form/SV_0ONZfm78WbSrNNI.

Bringing us onto a second type of bias…

• Was telling you exactly how the study works and what I’m trying to tell you the best way of conducting this survey?

• Probably not!

  • You might have wanted to help me with the demonstration by pretending to have used ChatGPT
  • Or you might not have really trusted that I can’t solve the fundamental problem of causal inference and thought this was a trap!

Measurement “problems” can be measurement opportunities

• Sometimes we can measurement “problems” as an opportunity to measure concepts of interest.
• This is especially true for research about social norms and social stigma.

Measuring social norms

Ch. 4 of Valentim (2024):

Measuring social norms