10: How to make research methodology choices

This chapter goes into the issue of differing levels of causal sophistication, including a quantitative methods flow diagram that can guide the reader through different approaches and what they might score on the evidence hierarchy. The chapter then describes several quasi-experimental research designs, from simple posttest only designs to the popular two-group pretest-posttest design. Qualitative research, mainly through structured, semi-structured, or unstructured interviews is covered, as well as focus groups and field observations. The chapter concludes with several survey question design examples.

Known error

There is a known error in figure 10-5 on page 141. It was introduced by the publishers and not caught in proof reading. Honestly, I have no idea why they would do this, but they did. Not only did they take two separate graphics lines and combine them, but they also changed wording on one bubble. My apologies for not catching it. The INCORRECT figure looks like this:

The following is the CORRECT version, the version that I sent the publishers.

Glossary terms in this chapter

Evidence hierarchy: A scale of quantitative research methods where larger scores indicate study methodologies that are likely to have stronger internal validity and value for policy makers looking to evaluate research studies.

Quasi-experimental design: Attempts to approximate the characteristics of a true experiment even though it does not have the benefit of random allocation of units to treatment and control conditions.

Natural experiment: Policy situations that occur in the real world without any researcher intervention that provide an opportunity for observational study.

Structured interviews: Structured interviews follow a pre-designed interview protocol used to guide the researcher. Structured interviews do not stray from the interview guide.

Semi structured interviews: Semi structured interviews start with an interview protocol as a guide but give the researcher an opportunity to probe the participant for additional details through the interview process.

Unstructured interviews: Tend to be the least guided and most conversational approach. Researchers use this to establish a rapport or to lead into sensitive subjects. The researcher has to take the lead.

Focus groups: Moderated meetings with a small group of participants who are carefully selected for their ability to contribute insights into the research topic.

Field observation: Involves a researcher directly gathering information from a police or criminal justice setting by observing participants in their community or work environment.

Participant observer: Researchers conduct themselves in such a way that they become an unobtrusive part of the scene, people the participants take for granted.

Additional information and links

The qualitative-quantitative divide

It might seem a little over-the-top that researchers get in disagreements about the researcher divide between qualitative and quantitative, but has professional repercussions. As this nice article from Scott Jacques summarizes, it is also potentially the case that if a quantitative and qualitative research article arrive at the same conclusion, the "quantitative version will be deemed more important, and the qualitative one as more attractive."

Correlation is not causation

The issue of correlation is not causation is a perpetual challenge in evaluation research, and a frequent cause of problems, because - frankly - the public are not well educated about causality. As this short article from The Guardian nicely puts it, "Our preconceptions and suspicions about the way things work tempt us to make the leap from correlation to causation without any hard evidence."

There is also a nice short summary from The Decision Lab where they note key figures, including someone that I tend to reference quite a bit, English polymath Sir Francis Galton.

Focus groups

The American Association of Nurse Practitioners has a one-page summary of focus groups. While if differs from my book in terms of how many practitioners to include in the focus group, it contains a number of useful considerations that should form part of your thinking about focus group work.

Surveys

If you want more information than in available from my brief book on survey research, then the survey company Qualtrics have a solid guide to different types of survey question design. It is useful because they incorporate styles of question (such as sliders) that really are only available for online format. Given how much we are moving online, this adds welcome content.

Related Reducing Crime podcast episode

Given this is a technical chapter, it is tough to get a podcast episode for this chapter that would not be horribly dry. As a result, here is a chat with Dr. Shon Barnes, the police chief for Madison, WI and a strong proponent of evidence-based policing. He discusses what it means for him as a decision leader in the organization.

#48 (Shon Barnes)

Shon Barnes is the police chief in Madison, Wisconsin. A new documentary (the 54th mile policing project) follows Chief Barnes and two other black police officers as they undertake a historic civil rights march from Selma to Montgomery Alabama. Jerry Ratcliffe chats with Dr. Barnes about his embracing of education and evidence-based policing, the challenges of working with communities in the post-George Floyd world, and the lessons he took away from his three-day trek across Alabama.