Reducing alcohol-related harm and social disorder in a university community: a framework for evaluation
- 1Injury Prevention Research Unit, Dunedin School of Medicine, Dunedin, New Zealand
- 2Department of Preventive and Social Medicine, University of Otago, Dunedin, New Zealand
- 3School of Medicine and Public Health, University of Newcastle, New South Wales, Australia
- Correspondence to Kimberly Cousins, Injury Prevention Research Unit, Dunedin School of Medicine, PO Box 56, 55 Hanover Street, Dunedin 9054, New Zealand;
Contributors KC, JC and KK developed the evaluation framework and methods of collecting the data. KC, JC and KK contributed to the writing of the manuscript. All authors read and approved the final manuscript.
- Accepted 13 May 2010
Background In New Zealand and other middle to high income countries, university student are at high risk of alcohol-related injury and other problems due to their typical pattern of episodic heavy drinking. In 2007, one university implemented Campus Watch, a novel and extensive programme to reduce social disorder, including alcohol-related injury, in the university area.
Objectives To quantify the effects of this complex intervention.
Setting A large public university campus and surrounding community in New Zealand.
Design A health promotion evaluation model was used, examining: (1) how the programme was developed, introduced and received by the community? (process); (2) whether the programme affected behaviour? (impact); and (3) whether the programme reduced social disorder and alcohol-related harm in particular? (outcome). The outcome phase uses a non-equivalent control group design to measure changes occurring in the Campus Watch area compared with other universities, and with a same-city control site.
Participants Programme staff, university students and other community members.
Data Interviews with university administrators and Campus Watch staff; surveys of local residents' views; Campus Watch incident data; national surveys of university students in 2005, 2007 and 2009; police data; fire department data.
Outcome Measures Prevalence of heavy episodic drinking; number of acute alcohol-related harms; incidence of antisocial behaviour, assault and street fires.
Analysis Regression analyses will be used to examine changes in the intervention site relative to changes in the control areas.
- public health
In New Zealand, 51% of alcohol-attributable deaths and 72% of life-years lost are due to injury.1 The total cost of alcohol-related harm in New Zealand is over NZ$2 billion annually. As in other developed countries, young people (aged 15–29 years) are at a particularly high risk of alcohol-related harm from their alcohol consumption, as they tend to drink to intoxication.1
The social and economic costs of alcohol are especially evident in North Dunedin (see figure 1), where university students comprise a large portion of the population. University students drink more frequently and more hazardously than their non-student peers.2–5 A tendency to drink large amounts per occasion increases the likelihood of experiencing acute harm, such as being physically or sexually assaulted.4 6–8 Secondhand effects of student drinking are also common among non-drinking students and extend into surrounding neighbourhoods.8–10
In 2006, the University of Otago in Dunedin, New Zealand, and the North Dunedin community created a working party to address the increasing social disorder in the area immediately surrounding the campus (see figure 1). Following the recommendations of the working party, the university launched the Campus Watch programme in 2007, which aims ‘to maintain and improve the quality of the student experience in the wider campus environment and to assist residents and businesses of North Dunedin with any concerns they may have’.11
Campus Watch is primarily concerned with reducing social disorder to which alcohol-related behaviour and harms are central. The programme provides round-the-clock foot patrols of the North Dunedin and campus areas. Most of the work done by the Campus Watch teams is described as pastoral care of students who are new to living in unsupervised shared housing, as well as relationship building with residents and businesses in the area. During the day, Campus Watch patrol teams focus on building rapport with students living in the area by chatting and offering advice or following up on previous incidents. They also act as a security patrol on campus and provide directions to visitors, as their distinct uniforms make them conspicuous in the university precinct (see figure 2). After dark, the Campus Watch teams continue their rounds of the campus and surrounding neighbourhoods, with the aim of remaining a visible, approachable presence and preventing situations from getting out of hand. The Campus Watch patrols also frequently walk students home late at night and check whether intoxicated students need assistance.
The university proctor's office manages the Campus Watch programme. Campus Watch members ‘on the beat’ liaise with headquarters, and are also in communication with the fire service and local police in order to share information about incidents, such as sightings of possibly criminal behaviour. When students are apprehended for behaving antisocially, they may be referred to the university proctor for disciplining under the University's Code of Student Conduct.
For each incident attended by Campus Watch, a team member completes a brief incident report form, which includes basic information about the time and location of the event, the type of event, how it was notified, details of the person(s) involved and whether they had been consuming alcohol, the outcome, and any further action needed. These forms are entered into a central database, and the proctor uses the reports to deal with students who have been referred to him.
Campus Watch has similarities with Neighbourhood Watch programmes, in which residents patrol their neighbourhoods and report suspicious behaviour to the police. It also incorporates the local concept of Māori wardens. Māori are the indigenous people of New Zealand, and the role of the voluntary wardens includes ‘discouraging crime on the streets, assisting in keeping our youth and people safe, while being compassionate of those in need […], to help, rather than to arrest people’.12
Campus Watch differs from these and other initiatives due to the nature of the student and campus environment in Dunedin. First, there is a very high density of students living in the area immediately surrounding the campus: there are 3300 residents per square kilometre in the university precinct shown in figure 1 and approximately 90% of them are enrolled in tertiary education.13 Second, 75% of university students come from outside of the Dunedin area to study,14 and are therefore away from the typically moderating influence of parents. Finally, the university's Code of Student Conduct gives the proctor power to discipline students for events occurring outside of the official campus boundaries.
The recently updated Medical Research Council framework for complex interventions emphasises the need for carefully designed, structured evaluations.15 By adapting a well-established health promotion framework to suit the complex Campus Watch programme, this quasi-experimental evaluation will measure specific outcomes and create an understanding of how Campus Watch contributed to any reductions in alcohol-related harm and social disorder. More specifically, the study aims to answer the following key questions: (1) how was Campus Watch developed, introduced and received by the community? (process); (2) how has Campus Watch affected behaviour? (impact); and (3) has Campus Watch reduced social disorder and alcohol-related harm in particular? (outcome).
The evaluation of Campus Watch comprises a traditional three-step health promotion evaluation model.16 It has been used to evaluate a variety of health promotion programmes such as community injury prevention,17 youth mental health awareness campaigns18 and community programmes for reducing youth smoking.19
The framework includes three evaluation phases in order to understand how a programme has been developed and implemented (process), what behavioural change has occurred since implementation (impact), and how effective the programme has been at achieving its primary objectives (outcome). The Campus Watch evaluation model is underpinned by specific questions for each phase, as illustrated in figure 3. The measurements used in each phase are described in more detail below.
The process evaluation will assess the way in which the programme was initially developed and implemented, how it has changed over time, how it is being delivered and its acceptability within the community. This phase of evaluation is particularly important in complex interventions.15 Campus Watch has undergone many changes, both structural and functional, since it was introduced in 2007; its evolution has not been guided by an overarching design and modifications have occurred for reasons that have not always been well documented.
The term ‘formative evaluation’ is sometimes used to describe an investigation of the way in which an intervention was developed, and is most useful in helping to improve a programme in its initial phases.20 The purpose of the process evaluation is to provide insight into why the intervention may or may not be effective and to document changes in the programme delivery, rather than to influence the initial phase of the programme in any way. As we were not involved in the development of the intervention, a formative evaluation would not have been an appropriate element in our model. Instead, relevant elements of a formative evaluation are incorporated into the ‘process evaluation’ by documenting the development of the programme by regular interviews with the programme director.
Specific information on the programme's development will be obtained from the university's director of student services, the university proctors and Campus Watch staff. Information about modifications to the programme and the motivation for these changes will be collected prospectively during the evaluation. Results from the 2008 and 2009 North Dunedin community surveys of residents and local business owners/managers (described in the next section) will give an indication of the community's knowledge and perceptions of Campus Watch.
Impact evaluation: community surveys and Campus Watch data
The impact evaluation will focus on the behaviours of North Dunedin residents and students to establish whether or not the programme is creating a culture of safety and reducing antisocial behaviour. The impact will be measured using surveys of randomly sampled North Dunedin residents and businesses in 2008 and again in 2009 (community surveys). Non-student residents will be oversampled to make up 50% of the residents' sample. The survey's aim is to elicit the views of residents and business owners/managers on problems in their community, the contribution of alcohol to these problems, and the impact that Campus Watch has had in the area. They will be conducted using a drop-and-collect method,21 which involves personally delivering questionnaires to selected participants and returning to collect completed questionnaires at an agreed date and time.
Incident data from the Campus Watch database (described above) will be used to monitor changes in student behaviour since 2007. Changes in the frequencies of various types of incidents will provide an indication of behavioural change in the community. Using this database, it will also be possible to determine whether the community is actively utilising Campus Watch by reporting incidents rather than waiting for Campus Watch to appear.
Outcome evaluation: campus and city surveys
The outcome evaluation will consist of two studies: one comparing students on the Dunedin campus with other New Zealand university campuses, and the other comparing North Dunedin (the student residential area around the campus) with South Dunedin (a mainly non-student residential area).
National survey: University of Otago Dunedin campus and other university campuses
Data on student alcohol consumption and its first and second-hand effects on students were collected at six university campuses in 2005 and eight university campuses in 2007, as part of the hazardous drinking project undertaken by the Injury Prevention Research Unit.22 In 2009, the web-based student drinking survey was replicated at the same eight university campuses that were included in 2007 (n=3300), which include the six campuses involved in 2005 (n=2550). By comparing the results from each survey, we will be able to measure changes in alcohol-related harm and disorder over time at each campus, and to see whether these changes have occurred similarly across all campuses. This will provide an indication of Campus Watch's impact on the North Dunedin area by comparing it with the other campuses that have not implemented Campus Watch. These three cross-sectional studies provide data from before Campus Watch was introduced (2005), in the first few months of Campus Watch (2007) and in the third year of operation (2009).
For the 2009 national survey, we invited up to 860 students per campus (eight campuses in total). This was determined by a sample size estimate used for the 2005 and 2007 national surveys, which was based on previous work23 and assumed a hazardous drinking prevalence of 60% with a 95% CI of ±5.6%, and a response rate of 70%, estimated conservatively from pilot research.24 The methods are described in more detail in another publication.25
With at least 415 participants per campus each year, it will be possible to estimate proportions to ±0.05 with 95% CI. It will also provide 80% power to detect relative differences between Otago and all other campuses in any given year of ±0.08 with 95% CI, and differences between any 2 years at Otago of ±0.10 with 95% CI. Multivariate logistic regression and general linear models will be used to compare proportions and continuous variables. A p value less than 0.05 will be considered statistically significant and all estimates will be presented with 95% CI.
City surveys: Dunedin campus area and South Dunedin
Data routinely collected by the New Zealand fire service and the New Zealand police regarding deliberately lit street fires (eg, couch burning), assaults, injury and offences that are commonly alcohol related will be analysed for North Dunedin and South Dunedin from 2005 to 2009. Changes over time in the two areas will be measured and compared. This comparison controls for variation in climatic conditions (for fires) and will be important in assessing whether any legislative, economic or other environmental factors operating in Dunedin may have had a general impact on alcohol-related harm and disorder that would not have been separable from Campus Watch effects when comparing the Dunedin campus with other campuses nationwide.
The University of Otago and the local community are impatient to see evidence of changes in student behaviour in North Dunedin.26 27 The evidence base for non-regulatory programmes to reduce community-level alcohol-related harm28 and other social disorder29 30 is very limited. Accordingly, the opportunity presented by the implementation of this substantial community liaison programme for developing a research evidence base for future policy making should not be wasted. In order to ascertain how much Campus Watch is contributing to any changes in North Dunedin, a comprehensive evaluation is necessary, even if it may take considerably longer than the university and wider community would like. By using the health promotion framework described here, we expect to be able to: (1) describe how the programme was implemented; (2) determine whether it has affected behaviour, and if so, to (3) understand the mechanisms by which it produced effects.
Given the complex politics of addressing social disorder and immense resource costs, it would not have been possible to use a randomised design to evaluate Campus Watch, as campuses would not have accepted random allocation to an intervention condition (even if the costs were met by a third party). It should be noted that we, the evaluators, were not involved in the conception or implementation of Campus Watch, we have no control over the setting in which the programme operates, and we have been careful to maintain independence from the programme director and staff. We should, however, be able to measure specific outcomes using non-randomised comparison groups measured over time. Non-randomised comparison groups have been used in the evaluation of other community-based initiatives, such as the UK Neighbourhood and Street Wardens Schemes,30 31 and these designs have a long history in behavioural science.32
There are a number of risks inherent in the methods we have adopted. First, the Campus Watch incident data are subject to variation in service delivery and to changes in reporting that may bias estimates of change over time.33 For example, the number of Campus Watch team members has changed since 2007, and on busy nights, due to workload, officers may report only the major incidents. Second, changes in the incident report forms in mid-2007 have also affected how certain incidents have been recorded in the database. While we have taken care to monitor the programme's implementation closely and regularly and to document changes, there remains a risk that important variations in protocols could influence estimates of intervention effects.
Another risk to the evaluation is error in the New Zealand police data. The police data, like the Campus Watch data, are subject to changes in service delivery, for example, putting more police on the beat can create the impression that crime has increased simply because incidents are more likely to come to police attention.33 The fire data should not have the same service delivery issues given that the fire service is engaged solely in responding to incidents, in contrast to police, whose role includes deterring crime. Fire service data may facilitate the production of intervention effect estimates with greater validity than the police data; however, they permit only a narrow range of outcomes to be assessed.
Without previous knowledge of the introduction of Campus Watch in 2007, we were not able to include more specific measures in the 2005 and 2007 national student surveys that might have more precisely evaluated the effectiveness of Campus Watch. By measuring outcomes such as changes in student alcohol consumption and the effects of drinking on individuals and peers from the national surveys, and by analysing the Campus Watch incident reports and police data, we aim to understand what role the Campus Watch intervention has had in any reduction in alcohol-related harm and social disorder.
It is also possible that the changes that university administrators expect as a result of Campus Watch will take longer than 3 years to occur and will therefore be missed by the proposed evaluation. In the unlikely event that there is no evidence of change within the first 3 years of implementation, the careful specification of the study protocol will permit the evaluation to be extended, including repetition of the community and national student surveys and collation of further police and fire service data.
Community-based initiatives to reduce alcohol-related harm, disorder and crime have been evaluated using various models. The measures most often used to evaluate these programmes were residents' perceptions of safety and fear of crime, as well as police data. Van den Eynde and colleagues34 used global performance measures to evaluate a 3-year community crime prevention programme, but found that they were not able to attribute changes specifically to the crime-prevention programmes using such measures. A systematic review of Neighbourhood Watch programmes that used police crime data found conflicting evidence as to their effectiveness in reducing social disorder and crime.29
Neighbourhood warden schemes that were evaluated in the UK used both police crime data and residents' perceptions of safety.30 Unfortunately, there was a significant association between residents' knowledge of a warden programme and their sense of security, which made it difficult to determine whether an increased sense of security was a result of programme awareness or a reflection of real crime reduction. We expect that this will also be an issue in the Campus Watch area, as staff are highly visible, and programme awareness would affect respondents' perceptions of crime and disorder in the North Dunedin community surveys. By analysing both the community survey responses and the data from the police and fire service, we should be able to differentiate between perceived and actual changes in harm and disorder. While the UK warden evaluations attempted to do this, many of the non-intervention comparison groups adopted their own warden schemes, thus becoming intervention communities themselves.
It is possible that our comparison university campuses may have adopted new programmes to reduce alcohol-related harm and disorder. We are not aware of any such programmes despite regular correspondence with the universities. By including numerous comparison areas—seven university campuses and one other Dunedin area—we expect to be able to control for alternative explanations for any observed changes in North Dunedin.
The Task Force on College Drinking of the National Institute on Alcohol Abuse and Alcoholism outlined effective strategies to reduce excessive drinking by college students.35 Interventions with evidence of effectiveness for college students included individually focused brief motivational enhancement, changing alcohol expectancies and social norms clarification. Environmental interventions, including campus–community coalitions to address drinking, were found to be effective in general populations and showed promise of being effective in college populations.
Long-term interventions with a broad environmental approach are rarely targeted at university students, even though the negative impacts of student drinking on the communities surrounding campuses can be significant and long running.36 37 Toomey and colleagues38 reviewed environmental interventions to reduce alcohol-related harm among college students, and found that restrictions on where alcohol was advertised, purchased and consumed were effective, while a number of multistrategy approaches had not been well evaluated. Many of these multistrategy approaches involved campus–community partnerships, but evidence for their effectiveness was lacking.
Evaluation of the large ‘A Matter of Degree’ programme, in which campuses took an environmental approach to reduce alcohol-related harms on campus, found that campuses exposed to extensive environmental intervention had significant decreases in alcohol use, harm and second-hand effects measures compared with non-intervention campuses.39 The high environmental intervention campuses focused on changing policies on campus as well as addressing student behaviour and alcohol use off campus.
As Campus Watch is also a high intervention programme, we may be able to find similarities with the ways in which the ‘A Matter of Degree’ and other environmental interventions have reduced heavy drinking and related harm. Notably, however, Campus Watch is unique in its focus on quality of life rather than drinking behaviour per se. Campus Watch does not specifically target alcohol access or promotion, or even try to limit the amount of alcohol consumed in North Dunedin; rather, it is attempting to change what is considered acceptable behaviour in a densely populated student area with few existing social controls. There are fewer legal controls than exist in campus environments in the USA, where a drinking age of 21 years can be used to regulate student alcohol use. In New Zealand, drinking per se is not illegal at any age and purchase is legal for those aged 18 years and over. It will be of value to know whether this broad-based approach is effective in reducing alcohol-related harm and disorder and improving the quality of life of North Dunedin residents.
What is already known on this subject
University students have a high prevalence of alcohol-related injury and other harms.
Strategies that modify the environment tend to be more effective than individually focused interventions.
What this study adds
We present a study protocol for a mixed method evaluation of Campus Watch, a multifaceted intervention seeking to reduce social disorder on a university campus and surrounding community in New Zealand.
Funding The Campus Watch evaluation is part of a PhD project that is funded by a HRC/ACC PhD career development award. Funding for the 2008 community surveys was provided by the Department of Preventive and Social Medicine, University of Otago. Funding for the 2009 community surveys and for the 2009 national survey was provided by the national discretionary grant fund of the Ministry of Health and by the Alcohol Advisory Council of New Zealand.
Competing interests None to declare.
Ethics approval This study was conducted with the approval of the University of Otago Ethics Committee.
Provenance and peer review Not commissioned; externally peer reviewed.