A review of the farm management tool Overseer has found it unfit for purpose.
The government has committed to improving Overseer, promising to develop a next generation of the tool or make additional tools available over the next year.
The SMC asked experts to comment on what options we might have for farm nutrient management in the future.
Professor Louis Schipper, environmental biogeochemist, University of Waikato, comments:
“There isn’t any simple easy statement about how to progress immediately and longer-term but we critically need a tool like Overseer to allow us to balance production and its impacts on the environment. Otherwise, we might end up regulating inputs of nutrients and potentially decreasing flexibility that farmers might have to manage their land. The tool needs to have the confidence of farmers and the public and be supported with hard data. We have some existing data collected through time but this will need to be supplemented with extensive research that helps the engine of any model work better across broad environments.
“The review identified issues with the underlying setup of the model and its inner workings with one focus being how nitrogen cycling is captured. We can name many of the steps of the nitrogen cycling involved and have a general understanding of the factors that control these rates. If we want to manage nitrogen losses from farming these processes need to be well captured whether developing a new model or adapting Overseer. However, I suspect that we simply do not know how fast or slow some of these nitrogen processes are going in soil across the varying farming environments of our country. Some processes are very well known like nitrogen fertilizer inputs and nitrogen export in milk but others like denitrification (conversion of nitrate to nitrogen gas) or immobilisation are extremely poorly quantified. This requires a greater quantification of basic soil processes which is not a large part of our current research agenda.
“Simply importing an overseas model is unlikely to be very useful without extensive modification to fit our rather unique farming systems and underpinned by data from Aotearoa New Zealand. However, the internal structures of these international models may allow us to make more rapid progress if underpinned by careful soil and water research on our farms.”
No conflict of interest declared.
Professor Troy Baisden, environmental scientist, University of Waikato; and President of the New Zealand Association of Scientists, comments:
This comment is excerpted from Professor Baisden’s blog.
“An MPI-commissioned peer review report has concluded that key aspects of the Overseer model used for water quality regulation are not fit for purpose. The Parliamentary Commissioner for the Environment (PCE) has described the report as “devastating”, while those responsible for the model defend it to users.
“The contrast confirms an undesirable situation I’ve previously framed by saying “Overseer is the best model we have, because it is the only model we have.” The problem evolves from a paradox those who develop scientific models have been aware of. Models that are heavily used can be very difficult to improve and modify as needs change. A model’s usability can be its own enemy when it comes to incorporating detailed science. And it is important to note that the review didn’t examine the usability and value of the Overseer user interface, and farm data collection.
“Those looking at the situation outlined by the peer review panel’s report on the Overseer model last week, will be wondering how such an array of problems and challenges can arise. Overseer has become essential to New Zealand’s water quality regulation with an emphasis on regions and catchments where nitrate problems exist. Overseer was designed for these environments: well drained soils over aquifers. By becoming a one-size fits all model supporting regulation, the Overseer model has captured focus for the entire country, including areas and situations the review finds are outside its core strengths and stated purpose.
“Can the “next generation Overseer” overcome the paradox that its dominant position, large user base and interface makes needed model improvements challenging? Achieving this, or other approaches that quickly reach the scale and quality needed will be difficult. Given the urgency, some may say it will be risky and duplicative to allow multiple approaches to replace Overseer. I argue the diversity of principles and spatial scales (from farms to regional and national policy), and timeframes of need establish different targets and necessitate development of some complementary and overlapping tools. We should beware of ever again placing so much weight on a single tool.”
No conflict of interest declared.
Dr James Lockhart, Senior Lecturer, School of Management, Massey University, comments:
“One of the root causes of this problem is that science typically makes marginal/incremental contributions. Thomas Kuhn challenged what has emerged as ‘normal science’ in his book, “The Structure of Scientific Revolutions”. Kuhn, arguably one of the most influential philosophers of modern science, pointed out that most science operates to extend an existing paradigm by the pursuit of filling gaps in knowledge.
“Hence with ‘Overseer’ value is seen to be created by making it better, and increasingly better over time. That the fundamental assumptions may (or may not) be flawed is unlikely to be questioned. I think the defence of Overseer by the CEO was valid – it is now being used for purposes for which it was not designed. Worse there is a pursuit of national standards, rules to which the individual farmers are being expected to comply – which makes Overseer’s task all the more improbable to succeed. This then creates a sunk cost problem, making change all the more difficult.”
Conflict of interest statement: “I am a user/subscriber to Overseer in Horizons (Manawatu Wanganui Regional Council)”
Dr Scott Larned, Chief Scientist – Freshwater & Estuaries, NIWA, comments:
“NIWA welcomes the independent science review report on the OverseerFM model.
“The review answered several important questions we and others have had about the model (including model structure and prediction uncertainty) and its suitability for specific applications, notably limit-setting and assessing compliance with regional rules and consent conditions.
“The use of Overseer for regulatory purposes has grown rapidly over the last 10 years, largely in response to the requirements of successive National Policy Statements for Freshwater Management and the absence of alternate tools. NIWA concurs with the conclusion of the Science Advisory Panel, that Overseer is not suitable for regulatory purposes such as limit setting or demonstrating compliance. Overseer may continue to be a useful tool for farm planning, including nitrogen management, and for preparing farm environment plans, which are mandatory under the Essential Freshwater package.
“The government response to the Science Advisory Panel’s review sets out multiple options for regulatory support, including a completely new nutrient loss model. We consider that several new, fit-for-purpose predictive tools may be needed, in lieu of a single monolithic model. New tools should:
- take advantage of the rapid growth in environmental and agricultural data generation,
- guide the collection of additional data to overcome known deficiencies,
- utilise recent advances in modelling, data science and technology, such as continuous water quality monitoring, and
- ensure that prediction uncertainty is quantified and made transparent to all users.”
Conflict of interest statement: “NIWA scientists contributed to the 2018 PCE review of Overseer. NIWA scientists have developed wetland and riparian modules for Overseer; these modules were not included in the Science Advisory Panel review.”
Anthony Pleasants, Adjunct Professor in Agricultural Analytics, Massey University; Dr Stephen Taylor, Senior Lecturer in Applied Mathematics, University of Auckland; and Dr Graeme Wake, Adjunct Professor of Industrial Mathematics, Auckland University of Technology, comment:
“The number of dairy cattle in Aotearoa New Zealand has doubled in the last two decades to over six million, thereby giving rise to increased farm intensification and capacity. This drove the need for higher grass yield produced by more and sometimes different composition of fertilisers to be used. Both the fertiliser run-off and the nitrogen in urea promotes microbial growth mostly in waterways and can contribute to poor health outcomes and downgrading of our environment. Finding strategies to minimise this impact is a priority for the whole country and should be underpinned by sound science.
“The essential ingredient here is the subtle complexities of interacting temporal effects and circumstantial changes like the weather, run-off, mitigation and dissipation into the catchment (soil and waterways), and the grazing strategy, The accumulation of nitrogen from the animal excrement and the fertiliser not absorbed by the pasture into the catchment areas is the result. Any decision-support model must incorporate all this together, with the inherent statistical variability in both inputs and outcomes. The methods are well known here and widely used internationally. They give bounds and estimates on the grazing system pattern to be used. However, the ingredients in the Overseer FM tool which has been very unwisely but widely used here up to now, is an overly simplistic approach not incorporating temporal “cause and outcome” effects, and system interactions. These are essential. The calculations therein fail to address the key dynamical issues central to the problem. Within Overseer FM there have been even more elementary fundamental computational errors reported.
“In Overseer the key science is not woven together as proper science would dictate. The report now released, suggests that the development of Overseer has, alarmingly, not received input or peer evaluation from any experienced independent system modellers for which New Zealand is quite well-endowed. Some of these scientists even offered to help as the idea was first mooted in the 1990s.
“Rules and procedures should be determined by accepted wisdom and be transparent to those making the rules and to those who must carry them out. Here the science system has grossly failed for Aotearoa New Zealand. This overdue investigation of Overseer is a most welcome step in addressing this unfortunate blot on our scientific practice. Action should follow to bring New Zealand up to better ecological standards, using the collective skills we have. Much better and very different methodologies are certainly needed in our opinion.”
Conflict of interest statement: “All three of us are currently working together in a wider interdisciplinary team from 2020 on a partially externally funded major catchment modelling model development that captures all the essential features.”
Dr John Roche, Kaitohutohu Mātanga Pūtaiao Matua | Chief Science Adviser, Manatū Ahu Matua | Ministry for Primary Industries, comments:
How does Overseer work to model nutrient flows within New Zealand farm systems?
“Overseer is a decision support system to help farmers and growers make property-level decisions that reduce the loss of nutrients to waterways. Overseer uses farmer and grower input (e.g., feed and fertiliser) and output (e,g., milk and meat) data throughout the year to understand the flow of nitrogen (and other nutrients) into, and out from, a farm system. The data collated are a very important part of understanding a property’s nutrient footprint.
“Overseer was originally designed to help with fertiliser and farm effluent decisions. However, by combining these nutrient flows with long-term average climate data, Overseer could then be used to estimate the risk of losing nitrogen. Most of the science underpinning Overseer’s estimates has focussed on nitrate leaching (i.e., the loss of nitrogen via sub-surface drainage through the soil), as nitrate is the predominant form of nitrogen lost from dairy pastures.”
Is it fit for purpose for farmers and for regional councils as a regulatory tool?
“This depends on the proposed purpose; Overseer has been an important tool to help farmers and growers understand how farm management factors affect nitrogen loss and for positive behaviour changes. For this reason, we want farmers to keep using it. Furthermore, it is a valuable tool to help farmers estimate their Greenhouse Gas footprint and plan a sensible fertiliser strategy.
“The Scientific Advisory Panel (SAP) that reviewed the Overseer model identified some key parts of Overseer that reduce their confidence that it can accurately estimate nitrogen loss and this calls into question its use for regulation. We can partition the panel’s criticisms into different pathways for nitrogen loss. The criticisms, for example, around nitrate leaching can be fixed and we can have confidence in Overseer under these circumstances. This will also add significant value for modelling deep rooting pasture-species, something that has been discussed for a number of years by farmers and their advisers. The criticism about overland flow and the loss of other forms of nitrogen are valid and we need to determine if farmers, growers and regional councils need a tool with this functionality and whether Overseer is the best tool doing this. This is part of our work programme for the year ahead.”
Are there any tensions between these two uses?
“There aren’t tensions between these two uses. Farmers and regional councils have the same goal – improving freshwater quality. Both farmers and regulators need confidence in the tools that they use. This review has pointed to some key areas for improving Overseer and Government is committed to developing the next generation of Overseer to give farmers and regional councils that confidence: that what they do will lead to improvements in freshwater quality.”
What are some of its limitations and how could it be improved?
“Firstly, it’s important to acknowledge that modelling nitrogen loss across the diverse landscape, soil types, climates, enterprises, and enterprise intensities of Aotearoa New Zealand is incredibly difficult. Secondly, Overseer was developed to visualise nutrient flows in pastoral agriculture, and primarily dairy. So, it was developed to focus primarily on nitrate.
“One of the reasons the SAP concluded that they would not have confidence in its use across all of Aotearoa New Zealand was because of this focus on nitrate-N. However, in our most over-subscribed catchments, nitrate-nitrogen is the predominant form of nitrogen entering waterways. Therefore, Overseer’s focus on this was correct. That said, the SAP has identified some key areas by which we can improve Overseer’s estimate of nitrate leaching to give us greater confidence.
“The SAP acknowledged that overland flow of non-nitrate forms of nitrogen is important in large parts of Aotearoa New Zealand. Therefore, because Overseer did not adequately address these different loss pathways, they would not have confidence in Overseers estimate of total nitrogen lost. This is important context, we should retain the positive attributes of Overseer that the panel identified.
“MfE and MPI are beginning work to understand whether regional councils need a tool like Overseer to model Overland flow losses of nitrogen and, if they do, what is the best way to do this. There’s some more work to do to understand the suitability of a redeveloped Overseer, or another tool, for the arable and horticultural sectors. These will be considered as part of the work programme being undertaken over the next 12 months.”