Measuring cost-effectiveness and identifying key barriers and enablers of lasting behavioural change in the cane industry
Led by: Prof Sharyn Rundle-Thiele, GU
Project Summary
This project will:
- Develop methodologies for assessing cost-effectiveness and for identifying key drivers of variation in cost-effectiveness in data-challenging situations
- Use developed methods to evaluate the cost-effectiveness of a range of completed programs for reducing nitrogen runoff from cane production in Reef catchments
- Within recently-completed and current programs, identify key barriers and enablers of:
a. Engagement
b. Lasting behavioural change
- Compare and contrast evaluation methods (targeted individual approach compared and contrasted with a within project stakeholder approach) to deliver an adaptive management evaluation tool for measuring engagement whilst programs are on-going.
Inform future investment decisions by helping to identify key characteristics of cost-effective projects that will successfully engage stakeholders to deliver lasting behaviour change.
Project Publications
Project Webinar – 4th December 2020
Project Video
Project Description
Over past decades a considerable number of programs have been implemented with the aim of improving the ecological health of the Great Barrier Reef. Many of these programs have focused on changing farming practice to reduce loads of catchment-sourced stressors, principally nutrients (primarily dissolved inorganic nitrogen (DIN)), fine sediment and pesticides which act to reduce Reef resilience [Reef 2050 Water Quality Improvement Plan, 2017]. Programs have also sought to incentivise land use changes – such as wetland conversion and wetland restoration – which can actively reduce nutrient, sediment and pesticide loads delivered to the Reef Lagoon.
Considerable quantities of data have been collected on these programs, tracking expenditures, detailing voluntary uptake across the agricultural sector and, to a lesser extent, recording outcomes in the Reef Lagoon. Periodic reporting suggests that uptake of voluntary programs in agriculture may be plateauing (Scientific Consensus Statement 2017), with potentially serious consequences for cost-effectiveness. In this context, now is an appropriate time to evaluate the cost-effectiveness of these different programs with regard to their impacts on Reef health, understand reasons behind program uptake (or not), and, where appropriate, understand the extent to which the different approaches have previously or can be expected to engender lasting cost-effective practice change that in turn enhances water quality. These types of evaluation are not straightforward, given that economic data on program investments are commercially sensitive, and that key actors in many locations and settings are suffering from considerable survey fatigue. In line with recommendations in the Reef 2050 Water Quality Improvement Plan (2017, p 20.) this project takes a reflexive stance, delivering a research project focussed on developing an assessment approach to inform future investment planning. The combination of cost-effectiveness assessment and social assessment methods (social surveys and Interactive Management Assessment (IMA) workshops) permits investors to understand which types of projects in which particular locations are most appropriate for cost-effectively engaging a broader cross section of the agricultural sector to deliver lasting behaviour change to enhance Reef water quality.
Cost-effectiveness assessment
Cost-effectiveness is typically estimated to evaluate the performance of existing programs and tracked through time to predict the likely total cost of achieving particular objectives from on-going programs. Cost-effectiveness can be assessed relatively easily where data on expenditures and outcomes are readily available and provide good geographic and socio-demographic coverage. Where data are scarce, spatially patchy and/or where data access is necessarily restricted by commercial confidentiality, cost-effectiveness assessment is more challenging. This project will use state-of-the-art approaches to identify and evaluate the following aspects of cost-effectiveness for programs focussed on improving Reef water quality by reducing nitrogen losses from sugar cane production:
- Direct estimation of cost-effectiveness of an agreed set of nitrogen-focused cane programs to date, and prediction of future cost-effectiveness of program continuation. Cost-effectiveness will be expressed in terms of physical outcome delivered per dollar invested (e.g. DIN load reduction to relevant sector of Reef lagoon per $ spent).
- Identification of key factors influencing cost and effectiveness, such as GIS-derived physical characteristics of project location (e.g. size, soil type, rainfall, slope), economic characteristics of the implementing agri-business (e.g. scale, cropping mix), and socio-demographic characteristics of the implementing landholder (e.g. age, experience, history of previous engagement with water quality initiatives). The variation in relative cost-effectiveness of program implementations across sites will be assessed using output distance function and data envelopment analysis (DEA) techniques (e.g. Bostian et al (2015), Whittaker et al. (2017)). Predictions of the DIN reductions achieved by implementing changes in management practice and land use at different locations will be obtained from the Paddock-to-Reef Calculator Tool, after obtaining advice from the relevant modelling experts.
- Bayesian network-based methods will be considered for constructing synthetic datasets which mirror the properties of commercially confidential data, but without any traceable links back to the businesses from which the initial commercially confidential data were collected. Methods will build on the work of Winkler (2004), Thibaudeau & Winkler (2002), Graham, Young & Penny (2008). This will be applied if actual economic data is not available.
- Spatial distribution of point-location data on effectiveness of outcomes (e.g. DIN load reduction from changes in management practice, DIN removal by land conversion to wetland) across catchments will be developed using geo-referenced surface approaches such as average shifted histogram (ASH) estimators (e.g. Whittaker et al. (2003).
Federal and Queensland funded projects to reduce nitrogen losses from cane land through changes in management practice and land use will be assessed following consultation with project stakeholders. An early objective of this project will be to consult with key stakeholders to determine exactly which suite of programs will be assessed – mindful that the desire to obtain a comprehensive assessment will have to be balanced against current evaluation projects that are in progress and very real concerns around survey fatigue amongst project participants. The potential list of programs for assessment includes:
1) Reef Rescue/Reef Programme/Reef Trust Phase III regional delivery model
2) Cane reverse tenders
3) QG RP20/RP161 intensive extension approach
4) MSF Sugar Project Uplift
5) Project Catalyst
6) QG Project Cane Changer
7) QG Wet Tropics MIP
8) Enhanced efficiency fertiliser project
Behavioural change assessment
Behavioural change is typically assessed using repeated measures. In practice, methods include social surveys that require land owners/managers to complete surveys at two or more points in time and/or observations on practices. Change is calculated by comparing performance across self-reported and/or observed measures across time. Calls for more a reflexive stance are apparent (Gordon and Gurrieri, 2014, p 262) and evaluations have been challenged to move from a “prove” to “improve” mentality ensuring that evaluations are focused on learning from the experience gained to understand what improvements are needed to extend program success (McHugh and Domegan, 2016). Evaluation practice may be enhanced by broadening focus beyond the individuals targeted for change to include within-program assessment involving all program stakeholders (Kennedy et al., 2017; Truong, 2014; Wymer, 2011).
Replicating and extending on the method applied in TF3.3 to evaluate RP161, an annual social research survey may be undertaken for agreed in-situ Reef programs in cane production targeting growing practice and/or land use change. Additional Interactive Management workshops will be conducted on all programs in Years 1 – 2. The following approaches will be used:
- Process and outcome evaluation [social surveys] will be undertaken annually. A survey administered in consultation with program stakeholders seeking land owner/manager views on targeted growing practice and/or land use practice change [outcome], experience with the program (including strengths, weaknesses and areas for improvement), attitudes towards targeted practices and program involvement, sustained practice change intention, willingness to recommend participation in this program and confidence to maintain targeted practices [process] will be administered. This will provide de-identified annual self-reported practices and enablers and barriers to sustained growing practice and/or land use change in Year 1 and change assessments in Years 2 and 3.
- Outcome evaluation [observation] using within program practice performance data will be included in outcome evaluations where access to de-identified data is permitted by program stakeholders. Triangulation of data permits self-reported practice measures to be considered in light of practice data observed.
- Working with all program stakeholders an Interactive Management Assessment (IMA) workshop will be hosted. An action mapping technique will be employed in IMA to identify factors preventing change. A matrix structuring process is used to develop structural hypotheses to map interdependencies between program stakeholders based on the consensus logic of the group. By undertaking a within-program, assessment, strategies can be implemented whilst the program is underway to overcome causally significant factors.
All programs included for this study will be selected in consultation with program stakeholders following a workshop on project commencement. The Department of Environment and Heritage Protection have agreed to provide up to five programs in cane for this project, which will be agreed upon by all project stakeholders prior to project commencement.
NESP 2017 Research Priority Alignment
- This project considers the social and economic value of investments aligning with NESP cross cutting issue number 2.
- Specifically this project aligns to Tropical Water Quality Priorities:
- Theme 1 by extending understanding of cost-effectiveness and aspects of lasting behavioural change resulting from practical improvements to land management practices and land use that are linked to improved outcomes for tropical water quality and ecosystem health (1.2).
- Theme 2 through identification of practical management actions capable of protecting and improving water quality on the Great Barrier Reef. The project outlined in this proposal evaluates actions that protect highest priority ecosystems (e.g. GBR) or reinstate severely impacted ecosystems (2.3).
Project Keywords
Cost-effectiveness; Program evaluation; Behaviour change; Barriers; Enablers; Engagement.
Project Funding
This project is jointly funded through GU, JCU and the Australian Government’s National Environmental Science Program.
