De-prosecution and death: A comment on the fatal flaws in Hogan (2022a)

Abstract

In a manuscript recently accepted for publication in Criminology & Public Policy, Hogan (2022a) presents results from a synthetic control method analysis that suggests de-prosecution in Philadelphia in the mid to late 2010s resulted in a large increase in the number of homicides that occurred in the city. In this comment, we point out several critical errors in the analysis that when corrected flip the direction of the effect and render the author’s estimated effect null. Our primary concerns include the short pre-intervention period, a failure to correct for imbalance over covariates in the synthetic control models, the use of homicide counts instead of rates as an outcome, an inaccurate description of the data used, and an inadequate explanation of data cleaning procedures including missing data. We reproduce the author’s results after correcting for these issues and find no effect of de-prosecution on homicide. Thus, these flaws are fatal to the author’s findings and therefore the study should not be used to inform criminal justice policy. Considering the author’s unwillingness to share their data and code, we call for a greater dedication to open science and reproduction/replication in criminology.

  • Initial manuscript received on 09/26/2022
  • Revised manuscript and response to reviews received on 09/06/2023
  • Reviewers
    • Lauren Schechter
      • Review received on 10/31/2022
      • Compensated with $75 honorarium