Study 1
Krevor, Grube, and DeJong (2017) conducted a randomized community trial to assess the impact of Mystery Shops on ID checking for age verification in off-premise alcohol establishments. A total of 24 communities in Oregon and Texas were matched on baseline rates of ID checking for sale of alcohol to young-looking “mystery shoppers.” Communities in each state with 15 or more off-premises alcohol sales licensees were selected with assistance from the Oregon Liquor Control Commission and the Texas Alcoholic Beverages Control Commission. Three baseline Mystery Shop inspections were conducted at a minimum of 11 randomly selected licensees in each of the 24 communities across the two states, to determine each community’s baseline pass rate and organize communities into high-, medium-, and low- compliance groups. The communities were randomly assigned to one of three study arms: 1) Direct Feedback Reports (8 communities, 99 licensees), 2) Community-Level Reports (8 communities, 102 licensees), and 3) Control/No Reports (8 communities, 97 licensees). Licensees in Arm 1, Direct Feedback, received direct feedback on staff ID-checking behavior and responsible retailing resources (RR); in Arm 2, Community-Level Reports group, licensees received periodic reports on aggregate results of ID-checking behavior and RR resources; and in Arm 3, the Control/No Reports licensees received no feedback or community-level reports on ID-checking behavior. The CrimeSolutions review of this study focused on the comparisons between the Arm 1, Direct Feedback treatment group and Arm 3, Control/No Reports group. The three groups showed very similar rates of ID checking at baseline. All licensees in the two treatment arms were mailed an announcement describing the study and notifying them that they might at some point soon be subject to Mystery Shops for the purpose of education but not enforcement.
After the 3-month baseline inspections, 10 Mystery Shop inspections were conducted at each establishment in Arm 1, Direct Feedback Reports, during months 4 through 13. In this study arm, inspection results were reported to staff of the 99 licensees in real time through green cards (that explained that an ID check had been correctly performed) or red cards (that indicated a failed ID check). Follow-up letters were also sent to establishment managers to describe strategies to promote staff ID checking going forward. In Arm 3, for the control group, 10 Mystery Shop inspections were conducted during months 4 through 13 at each of the 97 establishments in the 8 communities, but no feedback or community-level reports were given.
The outcome of interest was the ID-checking rates. To analyze the outcomes, study authors used a series of generalized linear mixed model logistic regressions of the Mystery Shop pass rates (0 = failed to check ID; 1 = checked ID) with the state as a covariate to control for differences between Oregon and Texas. Due to the large baseline difference in ID-checking rates between Oregon and Texas, the study authors conducted separate analyses to explore whether the interventions were differentially effective in the two states.
Study 2
Grube and colleagues (2018) used a cluster randomized cross-over design to assess staff ID-checking behavior at alcohol establishments in 16 communities in four states: 1) California, 2) Massachusetts, 3) Texas, and 4) Wisconsin. Study authors identified two “pairs” – or four communities – in each state that had a sufficient number of licensees for the study. The communities ranged in population from 50,000 to 250,000 and were ethnically and socioeconomically diverse. Using a computer algorithm, each pair of communities was matched as closely as possible based on population, numbers of retail outlets, median family income, percentage of population under age 18 years, and racial/ethnic composition. The random sample comprised 324 on-premises, alcohol-serving establishments (which included bars, taverns, clubs, and restaurants that received at least 40 percent of their revenues from liquor sales) and 313 off-premises retailers (which included liquor stores and other retailers such as grocery stores, convenience stores, or drug stores, that sold alcohol for consumption elsewhere) to be invited into the study. Of the eligible outlets in the selected communities, 557 participated in the study. Each participating establishment was notified three times in advance of the intervention with information about what the study would entail, reminding them that their participation was voluntary.
After the first three monthly Mystery Shops, communities within each pair were randomly assigned to an early intervention condition (n = 8) or to a delayed intervention condition (n = 8) using a computerized random selection procedure. In the early intervention group, ID-checking feedback was provided immediately after the fourth Mystery Shop and continued for five additional Mystery Shops; inspections prior to the fourth Mystery Shop served to assess pre-intervention baseline measures. In the delayed intervention group, establishments were given feedback after the tenth Mystery Shop attempt, with five subsequent monthly Mystery Shop inspections. The first nine purchase attempts for the Delayed Intervention group served as the pre-intervention baseline. Licensees received tailored reports on their ID-checking performance, as well as summaries of their state’s sales-to-minor laws, a set of recommended alcohol sales policies (for example, checking IDs for anyone who looks to be under 30 years old), a monthly report of the aggregate Mystery Shop results for their community, and other materials to help them work more effectively with staff on age verification, including “How to Talk to Employees about Mystery Shop Inspections, Employee Pledge, The Case for Responsible Retailing.”
The outcome of interest was staff age-verification behavior (i.e., whether ID checks had been performed). After each purchase attempt, the mystery shoppers documented whether age verification had been requested, coding “0” for failed to request ID, “1” for requested ID. To examine the effects, the study authors employed three-level fixed-effects hierarchical logistic regressions with repeated Mystery Shop observations (Level I) nested within the 557 outlets (Level II), nested within the 16 communities (Level III). Additional analyses explored whether Mystery Shop feedback differentially affected ID-checking at on- and off-premises outlets.