Program Profile: Mystery Shop Programs to Reduce Underage Alcohol Sales

This intervention seeks to increase staff ID checks for the sale of alcoholic beverages at licensed establishments, to help prevent sales to minors. The program is rated Effective. The results were mixed. One study found there was a statistically significant increase in age verification rates resulting from the intervention, whereas a second study found no statistically significant increases in ID-checking rates. Overall, the results suggest the intervention influenced ID-checking rates.

An Effective rating implies that implementing the program is likely to result in the intended outcome(s).

This program's rating is based on evidence that includes at least one high-quality randomized controlled trial.

This program's rating is based on evidence that includes either 1) one study conducted in multiple sites; or 2) two or three studies, each conducted at a different site. Learn about how we make the multisite determination.

Program Goals

Mystery Shop programs use young individuals of legal age to attempt to purchase age-restricted products, such as alcohol, to assess and provide feedback to alcohol outlet staff on their age-verification behavior. This is measured through staff requests for identification and refusal of sales to individuals who do not present an ID. The goal of Mystery Shop programs is to increase ID checking for the sale and service of alcoholic beverages, and to assist communities to reduce underage alcohol sales to adolescents/minors.

Program Components/Key Personnel

Restricting adolescent access to alcohol is important to reduce the many negative outcomes associated with underage alcohol use, including car crashes (Yi, Williams, and Dufour 2001) and damage to the developing brain (Spear 2002). Mystery Shop programs are designed to support this effort. These programs involve young, legal-age customers who attempt to purchase an age-restricted product (i.e., alcohol) to observe staff ID-checking performance. They provide feedback (in some cases, immediately) to clerks and servers of alcohol outlets or licensees about their performance, and reports to owners/managers on staff behavior.

Mystery Shop programs can serve as a proxy for law enforcement compliance checks, as the latter can be expensive to implement and labor intensive, and therefore not conducted frequently enough to ensure compliance (Erickson et al. 2013). The programs also serve to remind alcohol licensees and their staff of the consequences of illegal sales to minors. Unlike law enforcement inspections, Mystery Shop programs do not entail legal penalties for staff failing to check the ID of someone who may look young enough to warrant an ID check but is not legally underage. Mystery Shop programs also provide retailer education on responsible sales and service practices and can serve to encourage retailers to work to improve staff performance related to underage sale of alcohol.

In the Mystery Shop protocol, a male or female customer (or “mystery shopper” between the ages of 21 and 24) enters an alcohol retail outlet and attempts to purchase an alcoholic beverage. If the clerk or server requests and looks at their ID (e.g., a driver’s license), the mystery shopper presents him or her with a green card that explains that the check had been correctly performed. In the event the clerk or server does not request the mystery shopper’s ID or is willing to sell the age-restricted item without examining an ID, the mystery shopper presents the clerk or server and the manager on duty with a red card. This indicates a failed ID check and explains the possible penalties that would have occurred if this had been a law enforcement compliance check. A written follow-up report summarizing the Mystery Shop outcome is mailed to the store owner or manager, along with responsible retailing materials.

Study

Grube and colleagues (2018) used a cluster randomized cross-over design to assess staff ID-checking behavior at alcohol establishments in 16 communities in four states: 1) California, 2) Massachusetts, 3) Texas, and 4) Wisconsin. Study authors identified two “pairs” – or four communities – in each state that had a sufficient number of licensees for the study. The communities ranged in population from 50,000 to 250,000 and were ethnically and socioeconomically diverse. Using a computer algorithm, each pair of communities was matched as closely as possible based on population, numbers of retail outlets, median family income, percentage of population under age 18 years, and racial/ethnic composition. The random sample comprised 324 on-premises, alcohol-serving establishments (which included bars, taverns, clubs, and restaurants that received at least 40 percent of their revenues from liquor sales) and 313 off-premises retailers (which included liquor stores and other retailers such as grocery stores, convenience stores, or drug stores, that sold alcohol for consumption elsewhere) to be invited into the study. Of the eligible outlets in the selected communities, 557 participated in the study. Each participating establishment was notified three times in advance of the intervention with information about what the study would entail, reminding them that their participation was voluntary.

After the first three monthly Mystery Shops, communities within each pair were randomly assigned to an early intervention condition (n = 8) or to a delayed intervention condition (n = 8) using a computerized random selection procedure. In the early intervention group, ID-checking feedback was provided immediately after the fourth Mystery Shop and continued for five additional Mystery Shops; inspections prior to the fourth Mystery Shop served to assess pre-intervention baseline measures. In the delayed intervention group, establishments were given feedback after the tenth Mystery Shop attempt, with five subsequent monthly Mystery Shop inspections. The first nine purchase attempts for the Delayed Intervention group served as the pre-intervention baseline. Licensees received tailored reports on their ID-checking performance, as well as summaries of their state’s sales-to-minor laws, a set of recommended alcohol sales policies (for example, checking IDs for anyone who looks to be under 30 years old), a monthly report of the aggregate Mystery Shop results for their community, and other materials to help them work more effectively with staff on age verification, including “How to Talk to Employees about Mystery Shop Inspections, Employee Pledge, The Case for Responsible Retailing.”

The outcome of interest was staff age-verification behavior (i.e., whether ID checks had been performed). After each purchase attempt, the mystery shoppers documented whether age verification had been requested, coding “0” for failed to request ID, “1” for requested ID. To examine the effects, the study authors employed three-level fixed-effects hierarchical logistic regressions with repeated Mystery Shop observations (Level I) nested within the 557 outlets (Level II), nested within the 16 communities (Level III). Additional analyses explored whether Mystery Shop feedback differentially affected ID-checking at on- and off-premises outlets.

Study

Krevor, Grube, and DeJong (2017) conducted a randomized community trial to assess the impact of Mystery Shops on ID checking for age verification in off-premise alcohol establishments. A total of 24 communities in Oregon and Texas were matched on baseline rates of ID checking for sale of alcohol to young-looking “mystery shoppers.” Communities in each state with 15 or more off-premises alcohol sales licensees were selected with assistance from the Oregon Liquor Control Commission and the Texas Alcoholic Beverages Control Commission. Three baseline Mystery Shop inspections were conducted at a minimum of 11 randomly selected licensees in each of the 24 communities across the two states, to determine each community’s baseline pass rate and organize communities into high-, medium-, and low- compliance groups. The communities were randomly assigned to one of three study arms: 1) Direct Feedback Reports (8 communities, 99 licensees), 2) Community-Level Reports (8 communities, 102 licensees), and 3) Control/No Reports (8 communities, 97 licensees). Licensees in Arm 1, Direct Feedback, received direct feedback on staff ID-checking behavior and responsible retailing resources (RR); in Arm 2, Community-Level Reports group, licensees received periodic reports on aggregate results of ID-checking behavior and RR resources; and in Arm 3, the Control/No Reports licensees received no feedback or community-level reports on ID-checking behavior. The CrimeSolutions review of this study focused on the comparisons between the Arm 1, Direct Feedback treatment group and Arm 3, Control/No Reports group. The three groups showed very similar rates of ID checking at baseline. All licensees in the two treatment arms were mailed an announcement describing the study and notifying them that they might at some point soon be subject to Mystery Shops for the purpose of education but not enforcement.

After the 3-month baseline inspections, 10 Mystery Shop inspections were conducted at each establishment in Arm 1, Direct Feedback Reports, during months 4 through 13. In this study arm, inspection results were reported to staff of the 99 licensees in real time through green cards (that explained that an ID check had been correctly performed) or red cards (that indicated a failed ID check). Follow-up letters were also sent to establishment managers to describe strategies to promote staff ID checking going forward. In Arm 3, for the control group, 10 Mystery Shop inspections were conducted during months 4 through 13 at each of the 97 establishments in the 8 communities, but no feedback or community-level reports were given.

The outcome of interest was the ID-checking rates. To analyze the outcomes, study authors used a series of generalized linear mixed model logistic regressions of the Mystery Shop pass rates (0 = failed to check ID; 1 = checked ID) with the state as a covariate to control for differences between Oregon and Texas. Due to the large baseline difference in ID-checking rates between Oregon and Texas, the study authors conducted separate analyses to explore whether the interventions were differentially effective in the two states.

Mystery Shop inspections are conducted by the BARS Program (http://www.barsprogram.com/), which recruits and trains mystery shoppers for alcohol purchase attempts. The program targets clerks, servers, and owners/managers of various alcohol outlets. The Mystery Shop protocol requires that if the clerk on duty has waited on the mystery shopper (MS) in a prior inspection, the MS leaves the establishment and the inspection is scheduled for a later time.

In addition to feedback on ID-checking behavior, Mystery Shops can provide other supporting materials to help licensees work more effectively with staff on the importance of age verification, “How to Talk to Employees about Mystery Shop Inspections, Employee Pledge, The Case for Responsible Retailing” (Grube et al. 2018).

Subgroup Analysis

With regard to subgroup findings, Krevor, Grube, and DeJong (2017) conducted an analysis by state and found that while the interventions were each related to increases in the odds of ID checking, the effects were not statistically significant for either state alone. The direct treatment intervention arm did not increase ID-checking rates in Oregon. In Texas, ID-checking rates increased among all treatment groups from baseline to the end of the intervention period, but the increase was not statistically significant. Grube and colleagues (2018) conducted subgroup analysis by alcohol outlet type, on-premise or off-premise, finding that both outlet types significantly increased ID-checking rates during the intervention. The intervention interaction was not significant, indicating that the Mystery Shop feedback had a similar effect on ID-checking at both types of outlet.

These sources were used in the development of the program profile:

These sources were used in the development of the program profile: