CrimeSolutions Programs by the Numbers

A program is a specific set of activities carried out according to guidelines to achieve a defined purpose. CrimeSolutions' goal is to determine how effective is this program according to the most rigorous evaluation(s) available? This page helps show how we get there. See also CrimeSolutions Practices by the Numbers.

CrimeSolutions Review Funnel showing how the number of programs narrows at each step
The CrimeSolutions Review Funnel showing how the number of programs narrows at each step. (View larger image.)

From Articles to Ratings - The Program Funnel

How do we get to the programs that we rate? 

  1. Screen articles and reports   
  2. Identify programs that have been evaluated and screen those evaluations
  3. Send programs for review 
  4. Assign ratings and post to CrimeSolutions

Thousands of Articles Screened

Results of articles screened - 34902 screened out, 3096 programs identified, 1076 programs reviewed, 642 programs rated
Tens of thousands of articles have been screened to identify thousands of programs, out of which, hundreds of ratings have been assigned. (View larger image.)

While CrimeSolutions has rated hundreds of programs, tens of thousands of articles have been reviewed and screened. Programs are identified for potential inclusion on CrimeSolutions through literature searches of relevant databases, journals and publications, and nominations from the field.

Of the tens of thousands of articles reviewed

  • 8% resulted in an identified program.
  • 3% resulted in a review.
  • 1% resulted in a finding of inconclusive evidence.
  • 2% resulted in a rating.

From Programs Identified to Programs Reviewed

After programs are identified, research staff review program materials to determine whether the goals of the program fall within the scope of CrimeSolutions. To fall within the scope, the program must:

  • Aim to prevent or reduce crime, delinquency or related problem behaviors (such as aggression, gang involvement, or school attachment).
  • Aim to prevent, intervene, or respond to victimization.
  • Aim to improve justice systems or processes.
  • Target an offender population or an at-risk population (that is, individuals who have the potential to become involved in the justice system).

Prevention programs not explicitly aimed at reducing or preventing a problem behavior must apply to a population at risk for developing problem behaviors.

Results of program reviews: 434 inconclusive evidence, no effects 174, promising 376, effective 92
Programs Reviewed by Outcome (View larger image.) (View larger image.)

Historically, for every program identified:

  • 35% resulted in a review.
  • 21% resulted in a rating.
  • 43% were screened out.
  • 21% are put on hold.

For every program reviewed:

  • 40% resulted in a finding of inconclusive evidence.
  • 60% resulted in a rating.

For every program rated:

  • 27% resulted in a rating of No Effects.
  • 59% resulted in a rating of Promising.
  • 14% resulted in a rating of Effective.

Use filters to search for programs by rating, extent of evidence, and a number of other options.

Program Ratings by the Extent of Evidence - Single or Multiple Studies

Reasons for Findings of Inconclusive Evidence Reason	Number of Programs* Override option	16 Lacked Sufficient Info on Program Fidelity	133 Limited or Inconsistent Outcome Evidence	140 Inadequate Design Quality	145 *As of July 31, 2021  Inadequate Design Quality: studies do not provide enough information or have significant limitation in the study design (for example, small sample size, threats to internal validity, high attrition rates, etc.) such that it is not possible to establish a causal relationship t
Program Ratings: Single and Multiple Studies (View larger image.)

Program ratings can be based on up to three studies. The strongest evidence can come from programs that have undergone multiple evaluations. 

Of the programs rated:

  • 82% are based on a single study.
  • 18% are base on multiple studies.
  • 18% of programs rated No Effects are based on multiple studies.
  • 12% of programs rated Promising are based on multiple studies.
  • 56% of programs rated Effective are based on multiple studies.

Use filters to search for programs by rating, extent of evidence, and a number of other options.

Findings of Inconclusive Evidence - Why?

Reasons for Findings of Inconclusive Evidence
Reasons for Findings of Inconclusive Evidence (View larger image.)

Because we set a high-bar for the scientific rigor of the evaluations we use to rate programs, many evaluations do not meet our criteria and we are unable to rate the evaluated program.

Programs are included in the list of programs reviewed but not rated because of the insufficient research evidence about the program, not because of any known weaknesses in the programs themselves.

Program Ratings by Topic

Program ratings by topic
Program Ratings by Topic (View larger image.)

CrimeSolutions' literature review encompasses articles and reports addressing evaluations of programs across the criminal and juvenile justice systems and victim services. However, the availability of rigorous program evaluations is not equal across the eight primary topic areas covered by CrimeSolutions. 

As many programs address multiple issues across the justice system and victim services, each program may be assigned to multiple topics.

Visit the Topics page.

Program Ratings by Evaluation Type – RCTs or QEDs

CrimeSolutions uses rigorous research to inform you about what works in criminal justice, juvenile justice, and crime victim services. We set a high bar for the scientific quality of the evidence used to assign ratings — the more rigorous a study’s research design (e.g., randomized control trials, quasi-experimental designs), the more compelling the research evidence.

A randomized controlled trial (RCT) is an experimental research design in which participants are randomly assigned to a treatment or a control group. Most social scientists consider random assignment to lead to the highest level of confidence that observed effects are the result of the program and not the result of other variables.

Program reviews may include up to three evaluations, called the evidence base. To receive a “randomized controlled trial” tag on CrimeSolutions, at least of those studies must:

  1. Allocates groups via a valid random assignment procedure.
  2. Be rated highly for overall design quality by CrimeSolutions Study Reviewers.
  3. Have outcome evidence consistent with the overall program rating.

Programs tagged as RCTs may also include evaluations using quasi-experimental designs (QEDs) in the evidence base.

Of the programs rated on CrimeSolutions:

  • 47% include at least one RCT in their evidence base.
  • 62% of programs rated Effective include at least one RCT in their evidence base.
  • 33% of programs rated Promising include at least one RCT in their evidence base.
  • 71% of programs rated No Effects include at least one RCT in their evidence base.

Caption: Programs ratings are based on up to three evaluations. The blue portion of each bar represents programs that include at least one RCT in that evidence base. 

Program Ratings by Site Locations – Multi- vs. Single Site Evaluation

Programs implemented in different sites my show different levels of effectiveness for the same outcomes. Knowing a program has been evaluated in multiple sites can valuable to policymakers and practitioners considering implementing a program.

To receive a multisite tag on CrimeSolutions, a program must be evaluated 1) at more than one site within a single study; or 2) in more than one site across multiple studies. If the program is evaluated in more than one site across multiple studies, the studies’ ratings must be consistent (i.e., the demonstrated effects must be in a consistent direction) to receive the tag.

Of the programs rated on CrimeSolutions:

  • 17% are tagged as multisite.
  • 36% of programs rated Effective were evaluated in multiples sites.
  • 15% of programs rated Promising were evaluated in multiples sites.
  • 15% of programs rated No Effects were evaluated in multiples sites.

Top Ten Most Visited Programs

  1. The Bronx Defenders Holistic Defense Model (New York)
  2. Florida Work Release Program
  3. Allegheny County (Pa.) Jail-Based Reentry Specialist Program
  4. Mentally Ill Offender Community Transition Program (Washington)
  5. Adolescent Diversion Project (Michigan State University)
  6. Operation Peacekeeper (Stockton, Calif.)
  7. Big Brothers Big Sisters (BBBS) Community-Based Mentoring (CBM) Program
  8. Functional Family Therapy (FFT)
  9. Adults in the Making (AIM)
  10. Thinking for a Change

Program Rating Appeals

NIJ is committed to the quality, fairness and transparency of CrimeSolutions. As part of that commitment, you may inquire regarding an evidence rating or why a program has not been rated and posted to CrimeSolutions. Your inquiry should be grounded in:

  1. A fair reading of the evidence (please see the program's evidence base).
  2. An understanding of the rating procedures and criteria established for CrimeSolutions (please see How We Review and Rate a Program From Start to Finish or How We Review and Rate a Practice From Start to Finish).

Of the programs rated:

  • Twenty programs (3%) have been appealed.
  • Three program ratings (.5%) have been changed based on appeal and re-review. 
Date Created: September 21, 2021