Assessment

Table of contents

Background

Aims of the assessment

The Outcome Indicator Framework assessment provides consistent categories of historic change to help make comparisons across indicators and bring results together for a holistic picture across groupings such as individual 25 Year Environment Plan goals. Charts showing trends for individual indicators are useful to visualise how indicators have changed over time, but do not always simply show whether that change is occurring in a favourable direction aligned with policy ambitions.

The Outcome Indicator Framework assessment aims to provide high level patterns for key environmental indicators to identify where there may be challenges of progress to date in achieving goals. This will support the targeting of further investigation to establish whether additional action is required. It can also highlight where there may be informative examples of success. The assessment is intended to be of value for key high-level messaging on environmental change.

There are limitations to the insights that can be gained from the Outcome Indicator Framework assessment alone. These indicators were originally selected to show change over time and were not designed to assess progress towards any specific targets. The results of this assessment show where improvements have been made, but supplementary analysis would be required to judge whether such improvements are occurring at a fast enough pace or large enough scale to meet specific targets.

The Environment Act 2021 creates a new statutory cycle of monitoring, planning and reporting on environmental improvement, based around a long-term Environmental Improvement Plan. The 25 Year Environment Plan is the first such Environmental Improvement Plan. Separately from the annual updates to the Outcome Indicator Framework, government must report annually on what it has done to implement the Environmental Improvement Plan and on whether the natural environment (or particular aspects of it) has improved. That report will also consider the progress that has been made towards meeting targets.

The 25 Year Environment Plan was updated in 2023 with the publication of a new Environmental Improvement Plan. The Outcome Indicator Framework will undergo a review in 2024 to evaluate its ongoing suitability and consideration of best available evidence. This review will also consider the framework’s alignment with the updated policy landscape of the Environmental improvement Plan 2023. Current references to the 25 Year Environment Plan are maintained in advance of this review.

The Outcome Indicator Framework was designed to provide a robust account of changes in the environment. It was not designed to establish a causal link between an indicator’s observed trend and a specific driver of change (such as a policy intervention). Work to map indicator linkages for this assessment only highlights where indicators may be related. Further, detailed research would be required to prove the causes of any observed change in indicator trends.

The Outcome Indicator Framework assessment shows changes which have already occurred, but it does not make any predictions about whether these changes are likely to continue into the future. The likelihood of such future trends will depend on a wide range of matters including environmental factors, policy interventions which are yet to take effect, and changes in human activity which may have associated environmental impacts. More elaborated modelling research would be required to make forward projections accounting for the likely effect of potential or planned interventions. Such a forward projection on an entire 25 Year Environment Plan scale is not a planned ambition of the Outcome Indicator Framework. Detailed forward projections are produced for some indicators in the framework. For example, projections of UK emissions of air quality pollutants are compiled by Defra to inform policy development and to enable comparison with international commitments.

Method

Data availability

Of the 66 indicators currently included in the Outcome Indicator Framework, 10 are still in the process of being developed so could not be assessed at this point in time. Interim indicators were assessed in addition to final indicators as they can still offer useful insights. Some indicators currently show results for the latest year rather than a time series, reflecting current data availability of newly-developed indicators. As the assessment is based on trends, these could not be assessed. Some indicators do not yet have sufficient data points for a trend assessment across all time periods.

Data processing

Some indicators have multiple components (different related metrics, or different variables such as various pollutants). Each indicator component was assessed separately to avoid masking important results through aggregation and to allow individual consideration.

Some indicator components have a regular time series with a small number of irregular missing years of data. To allow trend assessment, missing years were extrapolated using the Excel fill function for a linear trend.

Most indicator data were smoothed before undertaking trend assessments to reduce the influence of natural interannual variability. The need for smoothing is demonstrated clearly by the D4b indicator for widespread butterflies in woodland which presents both smoothed and unsmoothed data. The unsmoothed data fluctuates widely year by year as butterfly abundance is very susceptible to changes in weather. If the first or last data point in the assessed time period happen to be an unusually good or bad year, this will skew the percentage change calculations. The smoothed values give users a more reliable indication of overall trend. A standard smoothing approach was adopted in most instances, as it is not practical to tailor smoothing for every indicator component individually. A Loess smoother was used as standard, as this is a versatile method suited to a wide variety of data.

Note that smoothed data represent modelled values while unsmoothed data represent the original recorded values. If there is a large amount of fluctuation in the time series, the smoothed values may be quite different from the original values. This means percentage change calculations based on smoothed data may also be quite different from what they would be if based on unsmoothed data. The decision on whether to use smoothed or unsmoothed data for individual indicator component assessments was made on a case by case basis, as deemed appropriate to suit the specific nature of the data series concerned.

Some indicators already produce smoothed data using a different approach selected specifically for that dataset. Where available, these existing smoothed values were used in the Outcome Indicator Framework assessment for consistency in messaging. Where indicators present a rolling average, smoothing was not considered necessary and the final year in the rolling time period was assigned to that data point. Smoothing was also not undertaken where there was no natural variability or error expected in the data (for example, values for the annual growth of trees that is harvested in E5 and the volume of timber brought to market in E6). Where smoothed data were used, the most recent data point was excluded before calculating the percentage change. Due to the smoothing process, the last data point in a series is often associated with greater error and deviates from the trendline to a greater degree, so including this more erratic value would lower confidence in the percentage change calculation. This is consistent with the approach taken elsewhere for assessment of some of the UK Biodiversity Indicators.

Where possible, trend assessments were undertaken for 3 time periods:

1. Short term - the most recent 5 years of change (6 data points),
2. Medium term - the most recent 10 years of change (11 data points), and
3. Long term - the whole time series (12 data points or more).

Some indicators have regular monitoring undertaken less frequently than every year; in these cases, an assessment was undertaken if at least 3 data points were available within the assessed time period. To provide clarity, the date range covered by each time period is specified for each indicator component and it is noted whether the percentage change values are based on smoothed or unsmoothed data.

There may be some instances where the long-term assessment only evaluates a small number of additional data points compared with the medium-term assessment. While the long-term assessment results represent change across the full available time series, care should be taken when comparing these results with the medium-term assessment results until there are at least 5 additional data points included in the former assessment compared with the latter.

An additional assessment was also undertaken to show change since 2018 (when the 25 Year Environment Plan was published) as this information is valuable for contextual consideration of environmental change. Data were treated slightly differently for this period as there were often only 2 or 3 years of data available. The unsmoothed data were always used and the last year of data included, so assessments could be undertaken for as many indicators as possible. This assessment gives some indication of changes since 2018 but it is not a comparably robust trend assessment to those based on the other longer time periods considered and results should be treated with caution.

Assessment of change

The assessment of change is based on the percentage change seen over the assessed time period, along with the desired direction of change for moving towards environmental goals.

First, it is established whether a ‘significant change’ has been observed. For most indicators, a threshold of at least 3% change (positive or negative) is used. This is consistent with the approach adopted for assessment of some of the UK Biodiversity Indicators and some other government assessments, such as Forestry Commission Key Performance Indicators. Where existing official Defra assessments were available for the same time periods using a more tailored methodology, these were replicated in the Outcome Indicator Framework assessment instead of applying the 3% threshold. The ‘little or no change’ category is intended to show indicators where any recorded change may be a result of random error in the dataset or due to chance, rather than a meaningful trend.

If a change has been observed, the direction of change is compared with the desired direction of change to assign a category of ‘improvement’ or ‘deterioration’. Some indicators do not have a desired direction of change (for example, E1 Area of agricultural land). These indicators may be provided for context to help understand other indicators, and additional information may be required to establish if an observed change is deemed to have a positive or negative connotation. Where a desired direction of change is not specified, indicators are assessed on whether change is significant.

Indicator links

A mapping exercise has been undertaken to show where indicators may be linked. Tables of potential links presented in the Summary of results section capture whether relationships between linked indicators are likely to be positive or negative and the rationale for including a link. The indicator mapping has been used to create network diagrams to highlight the interconnectivity between different environmental goals.

Identified indicator links are supported by text in the Outcome Indicator Framework itself, original indicator methods and assessment reports referenced in the ‘Readiness and links to data’ sections of indicator fiches, or any additional key references relating to individual indicator development provided by indicator leads. It was not deemed necessary for references to prove a statistical, causal link. Links simply show that the 2 indicators are likely to be related and it is possible a change in one may influence the results of the other. For the purposes of this exercise, there needed to be a direct link identified between specific indicators, not a chain of links culminating in an impact made by one indicator upon another indirectly-linked indicator. Links have been made based on the coverage of the indicator inferred by the indicator name. For example, D7 will eventually cover species supporting various ecosystem functions but for the immediate future it will focus on pollinators, so links were made on this basis.

Important considerations for interpretation of results

There are necessary time lags between data collection and presentation of results in the Outcome Indicator Framework. It is also not always possible to collect annual data for every indicator component. This means that the latest data point is not necessarily for the current calendar year and will differ across indicators depending on reporting frequency and time needed for data processing and analysis. Therefore, it is important to note the specific date range of the time period for each assessment result. This is provided in all the tables of assessment results in the individual indicator pages of this dashboard.

As outlined further in the method section, assessment categories are often assigned based on smoothed data. The percentage change values presented next to assessment of change categories in the tables of results will often be based on smoothed data. This means they may differ from values quoted elsewhere which are based on the raw, unsmoothed data, such as the trend description sections for individual indicators in this report. Tables of assessment results in the individual indicator pages of this dashboard always note whether percentage change values are based on smoothed or unsmoothed data.

Additional policy context is required to judge whether an assessment category is telling a positive story. A category of ‘improvement’ is clearly better than ‘deterioration’ but may not be positive if it is only a very small improvement for an indicator where are large amount of change is needed to meet an associated policy aspiration. Conversely, a category of ‘little or no change’ may not be a serious concern if any policy goals have already been achieved and current environmental status just needs to be maintained. A small number of indicators are slow moving, and change may not be expected to be observed in the short term. The narrative provided with indicator specific assessment results in the individual indicator pages highlights any important caveats such as these.

Summary of results