How We Rate Practices

Practices are identified for potential inclusion on CrimeSolutions through:

• Literature searches of relevant databases, journals and publications, including:
  • Social science databases using keywords identified in the areas of criminal justice, juvenile justice, and victims of crime;
  • Journals (including peer-reviewed journals) and other relevant resources; and
  • Other web-based databases of effective programs and meta-analyses of evaluated programs.
• Nominations from the field:   See how to nominate a practice here: Nominate a Program or Practice for CrimeSolutions.

For every article reviewed:

• 1.55% resulted in a meta-analysis identified.
• 0.45% resulted in a meta-analysis being reviewed.
• 0.02% resulted in a finding inconclusive evidence for the meta-analysis.
• 0.43% resulted in a meta-analysis contributing to a practice rating.

After practices are identified, research staff review practice materials to determine whether the goals of the practice fall within the scope of CrimeSolutions. To fall within the scope, the practice must:

• Aim to prevent or reduce crime, delinquency or related problem behaviors (such as aggression, gang involvement or school attachment);
• Aim to prevent, intervene or respond to victimization;
• Aim to improve justice systems or processes; and/or
• Target a population of persons convicted of a crime or an at-risk population (that is, individuals who have the potential to become involved in the justice system).

Prevention practices not explicitly aimed at reducing or preventing a problem behavior must apply to a population at risk for developing problem behaviors.

See a list of screened-out meta-analyses.

For every meta-analysis identified:

• 67% were screened out.
• 4% are on hold for future consideration.
• 29% were reviewed.

Based on the identified meta-analyses, CrimeSolutions Lead Researchers develop a clear definition that includes definite boundaries for what does and does not constitute the practice. Meta-analyses vary in scope such that some meta-analyses may address multiple practices or multiple populations. For example, CrimeSolutions often summarizes results separately for juveniles and adults, but both may be addressed within the same meta-analysis. The practice definition is used to develop search terms to identify additional meta-analyses, systematic reviews, and reviews in the practice area. CrimeSolutions does not limit the number of eligible meta-analyses that may be reviewed for any given practice.

All meta-analyses included in the evidence base for practices must meet the following minimum criteria:

1. Intervention. The meta-analysis must include at least two studies of the practice of interest.
2. Aggregation. The meta-analysis must aggregate the results from at least two studies.
3. Primary aim of the intervention. The programs included in the meta-analysis must aim to address the goals identified in the initial screening stage (Step 2 above).
4. Literature search. The literature search that guided the inclusion of primary studies in the meta-analysis must include at least two sources and must provide evidence that unpublished literature was sought in the search.
5. Primary outcomes. The meta-analysis must report on at least one eligible outcome related to crime, delinquency, overt problem behaviors (e.g., aggression, gang involvement, substance abuse), crime victimization, justice system practices or policies, or risk factors for crime and delinquency.
6. Control groups. All studies included in the meta-analysis must include an appropriate control, comparison or counterfactual condition, or the meta-analysis must analyze these studies separately from those that appropriate counterfactuals.
7. Reporting of results. The meta-analysis must report effect sizes that represent the magnitude of the treatment effect.
8. Combining effect sizes. When an average effect size is reported for multiple studies, all effect sizes in the combination must address the same type of relationship.
9. Publication date. At least 50 percent of the studies included in the meta-analysis must be published or otherwise available on or after 2000.
10. Age of samples. Samples included in the meta-analysis must be restricted to either adults or juveniles, or mean effect sizes for adults and juveniles must be reported separately.

Once the Lead Researcher selects the eligible meta-analyses that will comprise the practice’s evidence base, Study Reviewers begin the practice evidence review using the practice scoring instrument (access in PDF or Excel) to assess the quality, strength and extent to which the evidence indicates that the practice achieves its goals. Each meta-analysis within the evidence base is assessed by at least two Study Reviewers. The scoring instrument indicates the overall rating for each outcome that is reviewed.

See the Practices Scoring Instrument (PDF)

The scoring for practices consists of two parts looking at overall quality and internal validity:

Part 1: Overall Quality is assessed for each eligible meta-analysis included within the evidence base for a practice. The Study Reviewers make this assessment based on information about the methods and procedures used in each meta-analysis. Multiple items are scored to determine overall quality and these items are weighted differently based on their importance.

Study Reviewers assign numerical values to each element in the practice scoring instrument. The elements include a definition and other guidance reviewers consider when rating the elements. See the Practice Scoring Instrument (access in PDF or Excel) for guidance on each element.

Part 2: Internal Validity is assessed for each outcome within each eligible meta-analysis for a practice. Internal validity refers to the extent to which changes in the outcome can be attributed to the intervention, program or practice. Stated differently, internal validity refers to the extent that the research design is free from threats that may bias the estimated effect. There may be multiple outcomes that are independently assessed within each meta-analysis.

Randomized controlled trials (RCTs) have the strongest inherent internal validity. CrimeSolutions uses the extent to which average effect sizes are based on results from RCTs to assess internal validity. Meta-analyses based on a higher proportion of RCTs generally receive higher internal validity scores. Quasi-experimental designs may also be included in eligible meta-analyses for CrimeSolutions, but the internal validity ratings will generally be lower as the proportion of quasi-experimental designs increases. See the Practice Scoring Instrument (access in PDF or Excel) for guidance on how internal validity is scored.

To arrive at the outcome rating, the internal validity score for each outcome coded above is combined with the overall quality rating for the meta-analysis from which it was extracted. This information, along with information about the direction and statistical significance of the mean effect size, is used to categorize each outcome into one of following:

• Class 1 – Strong evidence of a positive effect
  Highest quality evidence with statistically significant average effect size favoring the practice
• Class 2 – Moderate evidence of a positive effect
  Moderate quality evidence with statistically significant average effect size favoring the practice
• Class 3 – Negative effect
  Moderate to high quality evidence with statistically significant average effect size in the opposite direction of the intended effect for the practice
• Class 4 – Non-significant or null effect
  Moderate to high quality evidence and the average effect size is not statistically significant from the comparison group
• Class 5 – Inconclusive evidence
  Limitations in the study design preclude CrimeSolutions from reporting further on these average effect sizes regardless of their direction or statistical significance.

Discrepancies among Reviewers: In the event that there is a classification discrepancy between the Study Reviewers, the Lead Researcher will work to achieve a consensus classification. If necessary, the Lead Researcher will also review the study and make a final determination on the classification.

For more information about scoring and classifying outcomes, see the Practice Scoring Instrument, access in PDF or Excel.

In cases where there is only one eligible meta-analysis for each outcome in a practice, the conversion from outcome classification described above to final outcome evidence rating is straightforward.

Outcome Evidence Rating Table

Outcome Evidence Rating	Icon [2]	Determination of Outcome Evidence Rating
Effective		Outcomes in Class 1
Promising		Outcomes in Class 2
No Effects		Outcomes in Class 3 or 4
Inconclusive Evidence	N/A	Outcomes in Class 5 – does not receive an icon

In some cases, multiple meta-analyses provide information about the same outcome. For each meta-analysis, that outcome is first rated separately for quality and validity as described above and then classified according to the five classes in Step 5 above. An additional step is needed to arrive at a final evidence rating for that outcome: Outcome findings are combined from multiple meta-analyses using the process below. CrimeSolutions does not limit the number of eligible meta-analyses that may be included.

Each class of outcomes is weighted as follows:

• Class 1 = 3
• Class 2 = 1
• Class 3 = -3
• Class 4 = 0
• Class 5 is not counted

Ratings for the same outcome from multiple meta-analyses are summed and averaged. The final evidence rating for the outcome is assigned based on this score.

• If the averaged points ≥ 1.50, then final outcome rating is Effective.
• If the averaged points are ≥ 0.50 and ≤ 1.49, the final outcome rating is Promising.
• If the averaged points ≤ 0.49, the final outcome rating is No Effects.

Outcome Evidence Rating Table

Outcome Evidence Rating	Icon [3]	Determination of Outcome Evidence Rating
Effective		The averaged points ≥ 1.50
Promising		The averaged points ≥ 0.50 and ≤ 1.49
No Effects		The averaged points ≤ 0.49

We consider re-reviewing a practice's rating when:

• New meta-analyses, or meta-analyses not previously identified, are found that meet the CrimeSolutions criteria. This may include meta-analyses that have been updated since the original review.

The new materials may or may not be sufficient to warrant a new evidence rating. If a Lead Researcher determines that there is sufficient evidence in the new materials to warrant another review, then the new information is sent to Practice Reviewers for assessment. Even if the outcome rating does not change, the new evidence and materials may be included or referenced on the profile page.

For more information: Inquiring About or Appealing an Evidence Rating.