Monitoring and evaluating transport investments can provide useful assessments of schemes indicating their performance against objectives and targets and, importantly, whether schemes are going as planned.
There are two types of evaluation typically included in the policy appraisal and evaluation cycle recommended by the Treasury, including
- Ex ante appraisal to decide whether a project should be implemented – often referred to as project appraisal.
- Ex post evaluation to assess the effects of a project after implementation.
While serious ex ante appraisals are obviously needed, ex post evaluations can be more important in the long run for influencing policy decisions. They should receive more attention and more resources than they do as they can produce hard evidence on outcomes and impacts, thus informing future ex ante appraisals, as well as capturing lessons learnt to improve the design of future projects.
For those areas that have secured Growth Deals, monitoring and evaluation will be required to:
- Ensure effective implementation.
- Find out ‘what works’ and where.
- Maintain the momentum around decentralisation.
- Support future bids.
- Provide accountability and transparency.
This message has been echoed in the findings and recommendations of the What Works Centre for Local Economic Growth’s Transport Review. The What Works Centre report highlighted the need for much more empirical work to be done on understanding the impact of infrastructure improvements on local economic growth. In particular the report recommendations focus on the need for more, and better, ex post impact evaluation and the need to embed such evaluation in the appraisal process.
The sections below provide an overview of the role of monitoring and evaluation and key guidance in undertaking successful monitoring and evaluation of transport investments.
Monitoring and evaluation are distinct activities: monitoring asks whether delivery is proceeding as planned and evaluation asks whether the intervention has achieved its desired objectives.
Carrying out monitoring and evaluation assessments is central to maximising the success of policies and schemes implemented by local decision-makers as well as to develop future policies that are effective and deliverable, applying lessons learnt from previous schemes.
In doing so, it is necessary to have clarity on how schemes will deliver the desired objectives, and to distinguish between outputs and outcomes. While outputs are the direct and immediate result of the implementation of a scheme – for example reduced journey times for bus users as a result of the introduction of a new bus priority lane – outcomes that follow from these outputs – such as economic benefits – are much more uncertain and can take time to be realised. For this reason, evaluating outcomes from infrastructure investment is a challenging exercise, as it is often difficult to unpick the outcomes from investment from other policies.
Logic maps provide a useful tool to monitor and evaluate scheme impacts. They are recommended by DfT and widely used in evaluation frameworks elsewhere. Logic maps provide a clear framework to understand how investments should deliver objectives. An example is provided below.
Monitoring and evaluation frameworks require a set of targets and metrics that are SMART (specific, measurable, assignable, realistic and time-related). These metrics should be based on the objectives for the investment – defined early on in the scheme development and reported in the Business Case. Overall, it is recommended that monitoring and evaluation frameworks are developed early on in the planning of infrastructure investments.
Finally, in addition to assessing outputs, impacts and outcomes against objectives, evaluation frameworks also look at the impact of the scheme compared to a hypothetical ‘Do Minimum’ scenario, in which the scheme was not implemented. For this assessment it is necessary to define a counterfactual scenario or reference case. This is a complicated exercise which may require the use of different forecast data sets and professional judgement. Ideally, the reference case should have been defined in the scheme Business Case when the economic analysis was produced.
In summary, while the monitoring of schemes provides useful feedback during the implementation phase allowing local decision-makers to respond to issues as they arise, ex post evaluations are vital to understand whether a project has achieved its targets after its completion. This provides key information for scheme promoters to maximise the success of current and future investments.