Following are some frequently asked questions associated with CrimeSolutions. You also may search a full list of questions and answers.
CrimeSolutions is intended to be a central, reliable resource to help practitioners and policymakers understand what works in justice-related programs and practices. Its purpose is to assist in practical decision making and program implementation by gathering information on specific justice-related programs and practices and reviewing the existing evaluation and meta-analysis research against standard criteria.
It is important to note the CrimeSolutions website does not constitute an endorsement of particular programs or practices. Furthermore, it is not intended to replace or supersede informed judgment and/or innovation. CrimeSolutions recognizes that rigorous evaluation evidence is one of several factors to consider in justice programming, policy, and funding decisions. NIJ also recognizes the importance of encouraging and supporting innovative approaches that may not yet have extensive evidence of effectiveness.
Read more about Tips for Using CrimeSolutions.
CrimeSolutions is operated by the National Institute of Justice. NIJ is the research, development, and evaluation agency of the U.S. Department of Justice.
No! CrimeSolutions is a resource to assist justice practitioners and policy makers in using evaluation evidence for practical decision making and program implementation. CrimeSolutions not intended to be an exhaustive list of worthy and unworthy investments. OJP and its components also recognize the importance of supporting innovative approaches and practices that may not yet have extensive evidence of effectiveness.
You are free to use the rating system and scoring instruments described and posted on CrimeSolutions. Information on CrimeSolutions, generated by the Department of Justice, is in the public domain and may be reproduced, published or otherwise used without our permission. When reproducing material from CrimeSolutions, including the scoring instrument, please acknowledge the National Institute of Justice and CrimeSolutions. When publishing or presenting work stemming from the use these materials, including programs ratings that you calculate, you must make it clear that such analyses, interpretations, and derivative work are yours and do not represent an actual or official rating from CrimeSolutions.
OJJDP’s Model Programs Guide (MPG), which predates the June 2011 launch of CrimeSolutions, focuses specifically on programs related to the juvenile justice system and covers the entire continuum of youth services from prevention through sanctions to reentry. CrimeSolutions serves a broader scope focusing on what works in criminal justice, juvenile justice, and crime victim services.
After the launch of CrimeSolutions, OJJDP began a re-review of MPG programs using the evidence standards and criteria developed for CrimeSolutions (see Program Review and Rating from Start to Finish and Scoring Instruments to learn more about this review process). Beginning in October 2013, the juvenile program information contained on CrimeSolutions and the MPG site will be completely aligned and will share program profiles. The existence of the two separate sites remains, though, because MPG also captures additional resources and tools that are specifically tailored to meet the needs of the juvenile justice field, including literature reviews and implementation guides.
OJJDP and the National Institute of Justice (NIJ), the sponsor of CrimeSolutions, are both components of the Office of Justice Programs (OJP), U.S. Department of Justice.
The table below lists several government and private organizations that maintain databases similar to CrimeSolutions and that identify what works in other disciplines.
The programs contained in these databases may or may not overlap with those found on CrimeSolutions. When a program on CrimeSolutions does appear in another database, we note that on the program's summary page.
|Program Library||Sponsoring Agency/Organization|
|Guide to Community Preventive Services||U.S. Department of Health and Human Services, Centers for Disease Control and Prevention|
|Model Programs Guide||U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention|
|National Mentoring Resource Center||U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention|
|What Works Clearinghouse||U.S. Department of Education, Institute of Education Sciences|
|Program Library||Sponsoring Organization|
|Blueprints for Healthy Youth Development (formerly Blueprints for Violence Prevention)||University of Colorado, Center for the Study and Prevention of Violence|
|Campbell Collaboration Library of Systematic Reviews||The Campbell Collaboration|
|Evidence-Based Policing Matrix||George Mason University, Center for Evidence-Based Crime Policy|
CrimeSolutions and these other clearinghouses are complementary resources. CrimeSolutions has a more general focus that includes all of criminal justice, juvenile justice and crime victim services. The clearinghouses each have differing but related or narrower focuses. As examples:
- The U.S. Department of Education's What Works Clearinghouse addresses the evidence base related to programs, products, practices and policies in education.
- The Model Programs Guide focuses on programs related to the juvenile justice system and covers all aspects of youth services, from prevention through sanctions to reentry.
In most cases, the clearinghouses use different systems and processes to rate their programs. The exceptions to this are the OJJDP-funded Model Programs Guide and National Mentoring Resource Center, which use the same rating methodology and instruments as CrimeSolutions.
Following are some available resources:
Being Smart on Crime With Evidence-Based Policing
In this National Institute of Justice NIJ Journal article from March 2012, Chief Jim Bueermann (Ret.) reflects on how law enforcement agencies can do a better job of using science to reduce crime by adopting evidence-based policing as a standard practice and by partnering with universities or colleges. Evidence-based policing leverages the country's investment in police and criminal justice research to help develop, implement and evaluate proactive crime-fighting strategies that may prove more effective and less expensive than the traditional response-driven models.
Is This a Good-Quality Outcome Evaluation Report? A Guide for Practitioners
This guide was produced by the Justice Research and Statistics Association and the Bureau of Justice Assistance to provide practitioners with a basis for distinguishing the differences between good- and poor-quality evaluation reports.
This free online resource developed by the Centers for Disease Control and Prevention offers practitioners, and others working to prevent violence, some important knowledge and resources for using evidence in their decision-making processes. This site is designed primarily for violence prevention practitioners, but anyone working to prevent violence in their communities will find the information useful.
Understanding Evidence, Part 1. Best Available Research Evidence: A Guide to the Continuum of Evidence of Effectiveness
In this guide, the Centers for Disease Control and Prevention aims to explain the purpose and meaning of the Continuum of Evidence of Effectiveness, a tool that was developed to facilitate a common understanding of what the Best Available Research Evidence means in the field of violence prevention. This Continuum also serves to provide a common language for researchers, practitioners and policymakers in discussing evidence-based decision-making.
Programs and Practices
The information and evidence ratings included on CrimeSolutions are not static. As additional programs and practices are identified and new research becomes available, CrimeSolutions content will be updated and supplemented to reflect the most current programmatic and research information available. We also rely on users to provide us with critical feedback about the CrimeSolutions website itself. What is useful and what is not? What additional features would you like to see on the site in the future? CrimeSolutions users are welcome to Submit Feedback.
Specific concerns about evidence ratings or information contained within CrimeSolutions may also be submitted via the Submit Feedback online form. If necessary, changes to the information presented on the site will be made, per the processes outlined in Inquiring About or Appealing an Evidence Rating.
CrimeSolutions does not provide direct services. However where available, we list contact information in the profile for each program and practice. If you are unable to locate this information on our site, please contact us with the title of the program or practice you are interested in, and we will do our best to provide the contact information you are seeking.
Although reviews are conducted on an ongoing basis, CrimeSolutions is not an exhaustive list of all justice-related programs and practices. If you cannot find the program or practice you are looking for, then you can do the following:
- Search for programs or practices that address similar outcomes using filters on the program and practice lists.
- Review the list of programs that have been considered but not rated . If the program is on this list, it means that the evaluation(s) associated with that program did not meet our criteria for review or inclusion.
- Review the list of practices that have been considered but not rated . If the practice is on this list, it means that the evaluation(s) associated with that program did not meet our criteria for review or inclusion.
- Nominate that program or practice in question for review.
Historically, for every article reviewed:
- 8% resulted in an identified program.
- 3% resulted in a review.
- 1% resulted in a finding of inconclusive evidence.
- 2% resulted in a rating.
For every program identified:
- 34% resulted in a review.
- 20% resulted in a rating.
- 45% were screened out.
- 20% are on hold.
For every program reviewed:
- 42% resulted in a finding of inconclusive evidence.
- 59% resulted in a rating.
For every program rated:
- 27% resulted in a rating of No Effects.
- 59% resulted in a rating of Promising.
- 14% resulted in a rating of Effective.
No. A listing on CrimeSolutions does not constitute an endorsement of particular programs or practices. Furthermore, it is not intended to replace or supersede informed judgment or innovation. CrimeSolutions recognizes that rigorous evaluation evidence is one of several factors to consider in justice programming, policy, and funding decisions. OJP also recognizes the importance of encouraging and supporting innovative approaches that may not yet have extensive evidence of effectiveness.
CrimeSolutions classifies programs and practices in three levels: “Effective,” “Promising” and “No Effects” based on the strength of the evaluation research that indicates a program or practice achieves its goals (i.e., its justice-related outcomes).
|Evidence Rating||One-Study Icon||Multiple-Study Icon||Description|
Programs or Practices have strong evidence indicating they achieve justice-related outcomes.
Programs or Practices have some evidence indicating they achieve justice-related outcomes.
Programs or practices have strong evidence that the program did not have the intended effects or had harmful effects when trying to achieve justice-related outcomes. While programs and practices rated No Effects may have had some positive effects, the overall rating is based on the preponderance of evidence.
A single-study icon is used to identify programs that have been evaluated with a single sample. A program with multiple publications listed in the evidence base may receive a single-study icon because:
- The publications resulted from a study based on a single sample.
- the studies that comprised the program’s evidence base did not demonstrate effects in a consistent direction.
A multiple studies icon is used to represent a greater extent of evidence supporting the evidence rating. The icon depicts programs that have more than one study in the evidence base demonstrating effects in a consistent direction.
Read more about evidence ratings at About CrimeSolutions.
A Program is a specified set of activities combined according to precise guidance in order to achieve a specific purpose. Program profiles on CrimeSolutions tell us whether a specific program was found to achieve its goals when it was carefully evaluated. The results apply to the exact set of activities and procedures used for that one program as it was implemented at the time of evaluation. Thus the program profile tells us that a program is likely to produce the observed result if implemented in exactly the same way.
A Practice is a general category of programs, strategies, or procedures that share similar characteristics with regard to the matters they address and how they do it. Practice profiles tell us about the average results from multiple evaluations of similar programs, strategies, or procedures. The programs, strategies, or procedures within a practice are similar because they share certain defining characteristics that are described for each practice profile on CrimeSolutions. Thus, practice profiles tell us the most typical results across multiple evaluations.
Sometimes a practice on CrimeSolutions will receive multiple evidence ratings because there is sufficient evidence available to draw conclusions about multiple outcomes addressed by that practice. This is because the ratings for practices are based on a larger amount of evidence using techniques of meta-analysis to examine the findings of numerous studies. With more evidence, it is possible to draw conclusions at a more detailed level of analysis. Programs on CrimeSolutions receive a single overall evidence rating because they are often based on only one evaluation study (although up to three may be included). With a smaller evidence base to inform conclusions, CrimeSolutions does not provide outcome-level ratings for programs.
CrimeSolutions includes “No Effects” programs and practices to inform policy makers and practitioners about the current status of available evaluation evidence before planning or implementing similar efforts.
No Effects programs and practices strong evidence that the program did not have the intended effects or had harmful effects when trying to achieve justice-related outcomes. In cases where negative effects were found, CrimeSolutions profiles for programs and practices will identify and describe observed negative effects.
While programs and practices rated No Effects may have had some positive effects, the overall rating is based on the preponderance of evidence.
Read more about Program Review and Rating from Start to Finish Tips for using CrimeSolutions.
You may notice that a practice's outcome ratings may not be the same as the programs that are encompassed by that practice and vice versa. For example, some practices have outcomes that are rated Effective but programs listed in the Related Programs section are rated Promising or No Effects.
This can happen because practice ratings reflect an assessment of the average effectiveness of a specific characteristic(s) common across programs, strategies, or procedures. Practice ratings do not take into account variations in implementation or other program-specific factors. If such variations impact the effectiveness of the essential practice characteristic(s), practice and program ratings may diverge as the program-specific effectiveness may differ from the average effectiveness across multiple programs.
A program with multiple publications listed in the evidence base may receive a single-study icon because:
- The publications resulted from a study based on a single sample.
- the studies that comprised the program’s evidence base did not demonstrate effects in a consistent direction.
The Review Process and Becoming a Reviewer
Yes. Programs and practices from other countries are included in CrimeSolutions. These must have English language citations and are otherwise subject to the same criteria used for programs and practices located within the United States.
We appreciate your interest in becoming involved with CrimeSolutions; but at this time, we are not looking for additional reviewers at this time.
For practices, Study Reviewers analyze all of the eligible and applicable meta-analysis studies to determine whether there is evidence that the practice achieves its goal(s). The Study Reviewers use a standard Scoring Instrument to determine the quality of each meta-analysis and to determine the internal validity of each outcome within the meta-analyses. The quality rating for each meta-analysis is combined with the internal validity rating of each outcome and the statistical significance of the effect size to determine a final evidence rating. If there are multiple meta-analyses that assess a single outcome, the individual outcome ratings from each meta-analysis are then aggregated to determine the final rating for that outcome.
For practices, individual evidence ratings are assessed for each outcome. Therefore, a single practice may have multiple evidence ratings if it addresses multiple outcomes. That is, one practice may be rated “Effective” for achieving one outcome, and “Promising” for achieving another outcome.
For programs, Study Reviewers analyze the most rigorous evaluation research available to determine whether there is evidence that the program achieves its goal(s). Up to three studies, representing the most rigorous evaluation research available, are selected to comprise the program’s evidence base. In some cases, a fourth study may be included in the review if Study Reviewers feel it is necessary to accurately assess a program. The reviewers use a standard Scoring Instrument for each study and assign scores across four dimensions:
- Program’s Conceptual Framework
- Study Design Quality
- Study Outcomes
- Program Fidelity
The individual study scores are then aggregated to determine the final evidence rating.
Read more about the Program Review and Rating from Start to Finish or see the Scoring Instrument.
Potential justice-related programs and practices are identified through:
- Literature searches of relevant databases, journals, and publications; and
- Nominations from experts, practitioners, or others.
Read more about how to Nominate a Program or Practice for CrimeSolutions.
Programs with evidence that is inconclusive as to whether they achieve justice-related outcomes do not receive an evidence rating on CrimeSolutions. However, CrimeSolutions periodically updates a static list of programs that have been reviewed by Study Reviewers, but not assigned an evidence rating due to lack of evidence. A program is placed on the inconclusive evidence list if the study (or studies) reviewed received only Class 5 study ratings indicating that there were significant limitations in the study design such that it was not possible to establish a causal relationship to the program’s justice-related outcomes.
A practice is placed on the inconclusive evidence list if the meta-analysis (or meta-analyses) reviewed received only Class 5 outcome ratings indicating there were significant limitations in the study design of the included evaluations that it was not possible to establish a causal relationship to the practice's justice-related outcomes.
Learn more about Programs and Practices Identified but Not Rated.
Before the evidence for a program or practice is reviewed and rated, the program or practice is screened to determine whether it meets CrimeSolutions' criteria for inclusion on the Web site. This includes a thorough review of the purpose of the program or practice and the strength of the available evidence. Read more about: Program Review and Rating from Start to Finish or Practice Review and Rating from Start to Finish.
At least two Study Reviewers independently assess the evaluation research for each program and the meta-analysis research for each practice under consideration by CrimeSolutions. All Study Reviewers are subject matter and research methodology experts who complete training and certification prior to reviewing studies. Lead Researchers select the evidence base, coordinate the review process, match programs with Study Reviewers based on expertise, and resolve any scoring discrepancies prior to a program being assigned a final evidence rating. Study Reviewers use a standard Scoring Instrument to determine whether a program achieves its goals.
Read more about CrimeSolutions Researchers and Reviewers or the Program Review and Rating from Start to Finish or the Practice Review and Rating from Start to Finish.
Using Information on CrimeSolutions
Intervention materials, including Web sites, may state that a program or practice has been reviewed and posted on CrimeSolutions. After a profile has been posted on CrimeSolutions, interested parties may Submit Feedback to request the CrimeSolutions logo graphic. The logo may be used as a link to CrimeSolutions or in the materials related to the program or practice. However, the posting of a program summary on CrimeSolutions does not constitute an endorsement, promotion, or approval of the intervention by CrimeSolutions or NIJ.
CrimeSolutions recommends the following citation format for program/practice profiles and summaries:
Office of Justice Programs, National Institute of Justice. Name of program or practice. Retrieved [month, date, year profile was accessed], from CrimeSolutions, [URL of summary].
No. However, we do provide contact information for the program developer, program director, training/technical assistance provider, and evaluator when available.
Please use the following text: This program [or practice] received the rating of ['Effective' or 'Promising'] by CrimeSolutions. For more information, visit [URL of CrimeSolutions program or practice profile].