CrimeSolutions Programs by the Numbers

A program is a specific set of activities carried out according to guidelines to achieve a defined purpose. CrimeSolutions' goal is to determine how effective is this program according to the most rigorous evaluation(s) available? This page helps show how we get there. See also CrimeSolutions Practices by the Numbers.

From Articles to Ratings - The Program Funnel

How do we get to the programs that we rate? 

1. Screen articles and reports   
2. Identify programs that have been evaluated and screen those evaluations
3. Send programs for review 
4. Assign ratings and post to CrimeSolutions

Thousands of Articles Screened

While CrimeSolutions has rated hundreds of programs, tens of thousands of articles have been reviewed and screened. Programs are identified for potential inclusion on CrimeSolutions through literature searches of relevant databases, journals and publications, and nominations from the field.

Of the tens of thousands of articles reviewed

• 8% resulted in an identified program.
• 3% resulted in a review.
• 1% resulted in a finding of inconclusive evidence.
• 2% resulted in a rating.

From Programs Identified to Programs Reviewed

After programs are identified, research staff review program materials to determine whether the goals of the program fall within the scope of CrimeSolutions. To fall within the scope, the program must:

• Aim to prevent or reduce crime, delinquency or related problem behaviors (such as aggression, gang involvement, or school attachment).
• Aim to prevent, intervene, or respond to victimization.
• Aim to improve justice systems or processes.
• Target populations of those who have committed crimes or are at-risk (that is, individuals who have the potential to become involved in the justice system).

Prevention programs not explicitly aimed at reducing or preventing a problem behavior must apply to a population at risk for developing problem behaviors.

Historically, for every program identified:

• 34% resulted in a review.
• 20% resulted in a rating.
• 45% were screened out.
• 20% are put on hold.

For every program reviewed:

• 42% resulted in a finding of inconclusive evidence.
• 58% resulted in a rating.

For every program rated:

• 27% resulted in a rating of No Effects.
• 59% resulted in a rating of Promising.
• 14% resulted in a rating of Effective.

Use filters to search for programs by rating, extent of evidence, and a number of other options.

Program Ratings by the Extent of Evidence - Single or Multiple Studies

Program ratings can be based on up to three studies. The strongest evidence can come from programs that have undergone multiple evaluations. 

Of the programs rated:

• 85% are based on a single study.
• 15% are base on multiple studies.
• 13% of programs rated No Effects are based on multiple studies.
• 11% of programs rated Promising are based on multiple studies.
• 33% of programs rated Effective are based on multiple studies.

Use filters to search for programs by rating, extent of evidence, and a number of other options.

Findings of Inconclusive Evidence - Why?

Because we set a high-bar for the scientific rigor of the evaluations we use to rate programs, many evaluations do not meet our criteria and we are unable to rate the evaluated program.

Programs are included in the list of programs reviewed but not rated because of the insufficient research evidence about the program, not because of any known weaknesses in the programs themselves.

Program Ratings by Topic

CrimeSolutions' literature review encompasses articles and reports addressing evaluations of programs across the criminal and juvenile justice systems and victim services. However, the availability of rigorous program evaluations is not equal across the eight primary topic areas covered by CrimeSolutions. 

As many programs address multiple issues across the justice system and victim services, each program may be assigned to multiple topics.

Visit the Topics page.

Program Ratings by Evaluation Type – RCTs or QEDs

CrimeSolutions uses rigorous research to inform you about what works in criminal justice, juvenile justice, and crime victim services. We set a high bar for the scientific quality of the evidence used to assign ratings — the more rigorous a study’s research design (e.g., randomized control trials, quasi-experimental designs), the more compelling the research evidence.

A randomized controlled trial (RCT) is an experimental research design in which participants are randomly assigned to a treatment or a control group. Most social scientists consider random assignment to lead to the highest level of confidence that observed effects are the result of the program and not the result of other variables.

Program reviews may include up to three evaluations, called the evidence base. To receive a “randomized controlled trial” tag on CrimeSolutions, at least of those studies must:

1. Allocates groups via a valid random assignment procedure.
2. Be rated highly for overall design quality by CrimeSolutions Study Reviewers.
3. Have outcome evidence consistent with the overall program rating.

Programs tagged as RCTs may also include evaluations using quasi-experimental designs (QEDs) in the evidence base.

Of the programs rated on CrimeSolutions:

• 47% include at least one RCT in their evidence base.
• 60% of programs rated Effective include at least one RCT in their evidence base.
• 32% of programs rated Promising include at least one RCT in their evidence base.
• 73% of programs rated No Effects include at least one RCT in their evidence base.

Program Ratings by Site Locations – Multi- vs. Single Site Evaluation

Programs implemented in different sites my show different levels of effectiveness for the same outcomes. Knowing a program has been evaluated in multiple sites can valuable to policymakers and practitioners considering implementing a program.

To receive a multisite tag on CrimeSolutions, a program must be evaluated 1) at more than one site within a single study; or 2) in more than one site across multiple studies. If the program is evaluated in more than one site across multiple studies, the studies’ ratings must be consistent (i.e., the demonstrated effects must be in a consistent direction) to receive the tag.

Of the programs rated on CrimeSolutions:

• 18% are tagged as multisite.

• 34% of programs rated Effective were evaluated in multiples sites.
• 15% of programs rated Promising were evaluated in multiples sites.
• 16% of programs rated No Effects were evaluated in multiples sites.

Program Rating Appeals

NIJ is committed to the quality, fairness and transparency of CrimeSolutions. As part of that commitment, you may inquire regarding an evidence rating or why a program has not been rated and posted to CrimeSolutions. Your inquiry should be grounded in:

1. A fair reading of the evidence (please see the program's evidence base).
2. An understanding of the rating procedures and criteria established for CrimeSolutions (please see How We Review and Rate a Program From Start to Finish or How We Review and Rate a Practice From Start to Finish).

Of the programs rated:

• Twenty programs (3%) have been appealed.
• Three program ratings (.5%) have been changed based on appeal and re-review.