NIJ is committed to the quality, fairness, and transparency of CrimeSolutions. As part of that commitment, you may inquire regarding an evidence rating or why a program has not been rated and posted to CrimeSolutions. Your inquiry should be grounded in:
- A fair reading of the evidence (please see the program's evidence base).
- An understanding of the rating procedures and criteria established for CrimeSolutions (please see How We Review and Rate a Program From Start to Finish or How We Review and Rate a Practice From Start to Finish).
Contact CrimeSolutions at [email protected] to submit an inquiry. If your inquiry is regarding a specific rating, we will provide additional explanation for the evidence rating that was assigned.
Note: We will not provide the identity of the reviewers.
Appeals Process
Following the inquiry and response described above, you may request a formal appeal of an evidence rating. Our goal is to complete the entire appeal process within 60 days.
We will work toward providing a systematic response that addresses all concerns.
Step 1. Contact CrimeSolutions at [email protected] to request a formal appeal of an evidence rating. You will then be provided additional guidance by CrimeSolutions, which will require that you provide pertinent, specific information that indicates the necessity for an appellate review.
Step 2. Two reviewers, selected on the basis of expertise pertinent to the appeal and the absence of any conflicts of interest, will conduct the appellate review. Each appellate reviewer will conduct an independent review of the appeal, the studies in the program's evidence base, and copies of the original reviewers' scoring instruments. The identities of the original reviewers will remain anonymous until they are required to participate in the process.
Step 3. After the separate reviews, the appellate reviewers will participate in a conference call to discuss the original review and rating, and the concerns raised by the inquirer, and to reach consensus about the program's evidence rating. If the appellate reviewers agree that the program's original rating should not be changed, they will provide a written explanation documenting the discussion and the reason for their final decision. If they believe a new rating is warranted, they will provide a written explanation describing how and why their scores differ from the scores of the original reviewers. A final conference call will be held between to discuss the disagreement on the scoring instruments and come to a consensus on a final program rating.
Step 4. Once a final consensus rating is reached, we provide a written response describing the final decision.
Step 5. If warranted as a result of the appeal, changes to a program's evidence rating or program profile will be made on CrimeSolutions. An appeal may also result in a re-review for the program.
Re-Review Process
We consider re-reviewing a program or practice when:
- New evaluation studies, or studies not previously identified, are found that meet the CrimeSolutions criteria. This may include studies that extend the follow-up period of previously reviewed studies.
- New supplemental materials are submitted that better explain the conceptual framework and fidelity dimensions of the program, which may affect a program's evidence rating.
- New meta-analyses, or meta-analyses not previously identified, are found that meet the CrimeSolutions criteria. This may include meta-analyses that have been updated since the original review.
The new materials may or may not be sufficient to warrant a new evidence rating. If a Lead Researcher determines that there is sufficient evidence in the new materials to warrant another review, then the new information is sent to the Study Reviewers for assessment. Even if the program's evidence rating does not change, the new evidence and materials may be included or referenced on the program's profile page.
If the re-review of a program results in a new evidence rating of Inconclusive Evidence, the program profile will be moved to the inconclusive programs list.